-- Logs begin at Thu 2022-09-29 01:31:22 UTC, end at Thu 2022-09-29 02:40:41 UTC. -- Sep 29 01:58:02.908828 dv-cinder-733062-36-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 29 01:58:02.920413 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: *** Starting uWSGI 2.0.18-debian (64bit) on [Thu Sep 29 01:58:02 2022] *** Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: nodename: dv-cinder-733062-36-iscsi Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: machine: x86_64 Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: clock source: unix Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: pcre jit disabled Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: detected number of CPU cores: 4 Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: current working directory: / Sep 29 01:58:02.923919 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: detected binary path: /usr/bin/uwsgi-core Sep 29 01:58:02.925991 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: your processes number limit is 47680 Sep 29 01:58:02.925991 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: your memory page size is 4096 bytes Sep 29 01:58:02.925991 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: detected max file descriptor number: 2048 Sep 29 01:58:02.925991 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: lock engine: pthread robust mutexes Sep 29 01:58:02.925991 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: thunder lock: enabled Sep 29 01:58:02.925991 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 29 01:58:02.925991 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 29 01:58:02.963136 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: Python main interpreter initialized at 0x558b8b6449c0 Sep 29 01:58:02.963136 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: python threads support enabled Sep 29 01:58:02.963136 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: your server socket listen backlog is limited to 100 connections Sep 29 01:58:02.963136 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: your mercy for graceful operations on workers is 90 seconds Sep 29 01:58:02.963485 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: mapped 403077 bytes (393 KB) for 2 cores Sep 29 01:58:02.963572 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: *** Operational MODE: preforking *** Sep 29 01:58:02.963814 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: *** uWSGI is running in multiple interpreter mode *** Sep 29 01:58:02.963814 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: spawned uWSGI master process (pid: 81670) Sep 29 01:58:02.964035 dv-cinder-733062-36-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 29 01:58:02.965047 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: spawned uWSGI worker 1 (pid: 81683, cores: 1) Sep 29 01:58:02.965047 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: spawned uWSGI worker 2 (pid: 81684, cores: 1) Sep 29 01:58:02.965047 dv-cinder-733062-36-iscsi devstack@keystone.service[81670]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 29 01:58:07.184178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:58:07.184178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: from cryptography.utils import int_from_bytes Sep 29 01:58:07.185000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:58:07.185000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: from cryptography.utils import int_from_bytes Sep 29 01:58:07.185438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:58:07.185438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: from cryptography.utils import int_from_bytes Sep 29 01:58:07.185556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 29 01:58:07.185556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: from cryptography.utils import int_from_bytes Sep 29 01:58:07.233628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 29 01:58:07.233833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 29 01:58:07.234020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] command line args: [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 29 01:58:07.234176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 29 01:58:07.234176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 29 01:58:07.234315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 29 01:58:07.234392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] command line args: [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 29 01:58:07.234470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ================================================================================ {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 29 01:58:07.234653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 29 01:58:07.234788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] admin_token = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.234868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ================================================================================ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 29 01:58:07.234930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] config_dir = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.235127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] admin_token = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.235265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.235265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] config_source = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.235404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config_dir = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.235482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] debug = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.235555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.235630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] config_source = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] debug = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] list_limit = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] log_config_append = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.237945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_config_append = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] log_dir = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] log_file = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_dir = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.238965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.239889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.240006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.240101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.240247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.240372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.240528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.240993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.61/identity {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.61/identity {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] publish_errors = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] publish_errors = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.241959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.242933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] standard_threads = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] standard_threads = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] use_journal = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_journal = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] use_json = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.243996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_json = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.244074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] use_stderr = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.244151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_stderr = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.244222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] use_syslog = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.244292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] use_syslog = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.244415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.244500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 29 01:58:07.244662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.244739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.244892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.244970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.245934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246108 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.246902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.247984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.248946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.249987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.250935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.251873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252479 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.252951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.253950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.254953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.255979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.256572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.256660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.256783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.256890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.256987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.257943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.258969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.259901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260814 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.260938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.261936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.262904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.263970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.264959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265589 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.265863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.266958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.267930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.268949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.external = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] auth.external = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.269908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.password = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] auth.password = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] auth.token = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] auth.token = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.270908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.271054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.271146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.271301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.271384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.271522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.271646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.271895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] credential.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.272998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.273950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.274904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.275901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.276966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.277888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.278915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.279976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.280969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.281876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.282999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283803 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.283942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.284927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.285844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.286939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.287991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.288139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.288220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.288371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.288451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.288641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.288721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.288951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.289964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.290929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.291855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.292943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.293926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.294897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.295980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.296983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.297909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.298843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.299918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.300000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.300161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.300239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.300396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.300475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.300622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.301984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] role.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] role.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.302968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.303899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.304857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305713 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.305952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.306053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.306216 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.306364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.306599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.306690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.306830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.306908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.307866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.308948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.309964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310604 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.310906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.311994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.312949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313421 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.313975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314237 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.314992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.connection = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.315922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.316982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.317981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.318151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.318247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.318392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.318499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.318684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.318764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81684) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 29 01:58:07.319716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 29 01:58:07.319879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81683) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 29 01:58:07.348901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.349053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.349133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.349215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.349336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.349427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.357624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.357797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.385723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.385877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.385947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.386032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.438684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.438873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.486208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.486427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.486959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.487200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.645006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.645218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.645435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.645668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.645844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.646108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.646976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.647466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.647721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.648094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.648480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.648852 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.669195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.669432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.669533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.669614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.669700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.669859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.669985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.670082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.670160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.670256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.670323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.670497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.670624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.670693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.681916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.682332 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.694642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.694783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.694863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.694940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.695868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81684) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.696024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81683) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 29 01:58:07.712498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.712752 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:58:07.712936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.713034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.713101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:58:07.713206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:58:07.713455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.713580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.713649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 29 01:58:07.713712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.713926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.714097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.714185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.714249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.714477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.714727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.723714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.723993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.724255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.724381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.724536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.724664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.724851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.725133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.731512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.732165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.734414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.734699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.734985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.735110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.735265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.735390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.735561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.735701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.735772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.735847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.736076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.736201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.736366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.736490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.736583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.736669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.736884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.737022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.737119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.737208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.737412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.737673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.760104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.760299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.761547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.761741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.761868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.762030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.766493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.766738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.766863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.766932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.766995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.767233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.773406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.773572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.773665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.773889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.779335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.779483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.779630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.779755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.779921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.780997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.781086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.781297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.781426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.796668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.796815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.796887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.796951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.801018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.801146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.804297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.804424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.804591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.804718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.808742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.808887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.811816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.811942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.815006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.815132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.815306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.815431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.815580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.815705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.815861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.815999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.816106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.816196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.816412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.816540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.832158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.832395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.832527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.832684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.832810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.832913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.833074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.833158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.839790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.840223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.841298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.841746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.842829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.843098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.843304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.843557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.845711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.845993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.846295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.846420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.846570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.846696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.846903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.847031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.847183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.847562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.858177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.858460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.858737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.858862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.859011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.859152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.859378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.859503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.859656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.859781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.859930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.860112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.860311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.860554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.875414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.875987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.878317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.878903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.881312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.881865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.882778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.883382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.884230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.884508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.884821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.884946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.885099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.885487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.891204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.891846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.892638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.893281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.895525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.895917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.896181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.896306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.896422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.896742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.896863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.897118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.904412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.904702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.904895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.905040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.905208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.905490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.911812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81683) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.912094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.912421 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81684) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 29 01:58:07.912546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.912772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.912879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.912946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.913101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.913240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.913353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.913444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.913673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.913847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.913952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.914060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.914185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.914278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.914455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.914551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.914744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.914865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.915001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.915100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.915260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.915350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.915529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81683) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.915618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.915800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81684) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 29 01:58:07.949856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x558b8b6449c0 pid: 81683 (default app) Sep 29 01:58:07.949980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x558b8b6449c0 pid: 81684 (default app) Sep 29 01:58:07.953080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3920c66f-41ad-43ba-91bf-914b0cea18a5 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:07.953360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3920c66f-41ad-43ba-91bf-914b0cea18a5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:07.953626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3920c66f-41ad-43ba-91bf-914b0cea18a5 None None] PATH_INFO: `/v3` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:07.957752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1/1] 10.222.0.61 () {50 vars in 791 bytes} [Thu Sep 29 01:58:07 2022] GET /identity/v3/ => generated 255 bytes in 7 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:14.778805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d9d8c3-9ef5-44d8-8258-e28e72c895c2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.779626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d9d8c3-9ef5-44d8-8258-e28e72c895c2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.779626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d9d8c3-9ef5-44d8-8258-e28e72c895c2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.786717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1/2] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:14 2022] GET /identity => generated 270 bytes in 14 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:14.793856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:14.794341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:14.794481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:14.875773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: INFO dbcounter [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] Registered counter for database keystone Sep 29 01:58:14.896824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG oslo_db.sqlalchemy.engines [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81683) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 29 01:58:14.905465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writer thread running {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 29 01:58:15.093486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG passlib.handlers.bcrypt [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81683) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 29 01:58:15.093866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG passlib.handlers.bcrypt [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81683) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 29 01:58:15.155575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:15.157925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:15.306145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 2/3] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:14 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 514 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 29 01:58:15.770937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d9d8c3-9ef5-44d8-8258-e28e72c895c2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:15.844700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: INFO dbcounter [None req-70d9d8c3-9ef5-44d8-8258-e28e72c895c2 None None] Registered counter for database keystone Sep 29 01:58:15.868958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG oslo_db.sqlalchemy.engines [None req-70d9d8c3-9ef5-44d8-8258-e28e72c895c2 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81684) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 29 01:58:15.880361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writer thread running {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 29 01:58:16.005907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70d9d8c3-9ef5-44d8-8258-e28e72c895c2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.013810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbb5a1d9-93f2-4fce-baf7-b17c0f9a5714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.014893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb5a1d9-93f2-4fce-baf7-b17c0f9a5714 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.015363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb5a1d9-93f2-4fce-baf7-b17c0f9a5714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.015804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbb5a1d9-93f2-4fce-baf7-b17c0f9a5714 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.027324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb5a1d9-93f2-4fce-baf7-b17c0f9a5714 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.079010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING keystone.common.rbac_enforcer.enforcer [None req-dbb5a1d9-93f2-4fce-baf7-b17c0f9a5714 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 29 01:58:16.079329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbb5a1d9-93f2-4fce-baf7-b17c0f9a5714 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.089673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 2/4] 10.222.0.61 () {58 vars in 1324 bytes} [Thu Sep 29 01:58:15 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 320 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 01:58:16.098900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:16.117785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5b4bcd-fd31-40ed-b63a-6a736cf79751 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:16.122803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8bca1e0-c971-4ea8-91c5-e4f4886b06ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:16.123908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bca1e0-c971-4ea8-91c5-e4f4886b06ab admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:16.124188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bca1e0-c971-4ea8-91c5-e4f4886b06ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:16.124465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8bca1e0-c971-4ea8-91c5-e4f4886b06ab admin admin] PATH_INFO: `/v3/projects/c6a6000fb0914b0390fa25e08205ee14` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:16.126087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bca1e0-c971-4ea8-91c5-e4f4886b06ab admin admin] RBAC: Authorizing `identity:get_project(project_id=c6a6000fb0914b0390fa25e08205ee14)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:16.178904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING keystone.common.rbac_enforcer.enforcer [None req-c8bca1e0-c971-4ea8-91c5-e4f4886b06ab admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 29 01:58:16.179269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8bca1e0-c971-4ea8-91c5-e4f4886b06ab admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:16.183787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 3/5] 10.222.0.61 () {56 vars in 1207 bytes} [Thu Sep 29 01:58:16 2022] GET /identity/v3/projects/c6a6000fb0914b0390fa25e08205ee14 => generated 344 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:18.365560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18537206-4226-47e9-b063-384f24d00e7c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.366334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18537206-4226-47e9-b063-384f24d00e7c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.366499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18537206-4226-47e9-b063-384f24d00e7c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.370716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 3/6] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:18 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:18.376702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c05f22-f3f7-4cb2-beec-d0b1489a703f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.377116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c05f22-f3f7-4cb2-beec-d0b1489a703f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.377201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c05f22-f3f7-4cb2-beec-d0b1489a703f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.399666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-06c05f22-f3f7-4cb2-beec-d0b1489a703f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:18.401168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-06c05f22-f3f7-4cb2-beec-d0b1489a703f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:18.409972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 4/7] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:18 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 29 01:58:18.680433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18537206-4226-47e9-b063-384f24d00e7c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.699214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18537206-4226-47e9-b063-384f24d00e7c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.703621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789316e4-d8eb-4257-a05c-37da57780533 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.704226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789316e4-d8eb-4257-a05c-37da57780533 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.704501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789316e4-d8eb-4257-a05c-37da57780533 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.704771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789316e4-d8eb-4257-a05c-37da57780533 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.713482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789316e4-d8eb-4257-a05c-37da57780533 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.715308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789316e4-d8eb-4257-a05c-37da57780533 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.724634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 4/8] 10.222.0.61 () {58 vars in 1324 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 01:58:18.731156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c05f22-f3f7-4cb2-beec-d0b1489a703f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:18.740901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c05f22-f3f7-4cb2-beec-d0b1489a703f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:18.746597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5313e4-43eb-45e8-ac68-9fb1d05e6704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:18.747224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5313e4-43eb-45e8-ac68-9fb1d05e6704 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:18.747497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5313e4-43eb-45e8-ac68-9fb1d05e6704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:18.747771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5313e4-43eb-45e8-ac68-9fb1d05e6704 admin admin] PATH_INFO: `/v3/users/4aa345955ed042db9837b36c3d6532b0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:18.749220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5313e4-43eb-45e8-ac68-9fb1d05e6704 admin admin] RBAC: Authorizing `identity:get_user(user_id=4aa345955ed042db9837b36c3d6532b0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:18.751092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5313e4-43eb-45e8-ac68-9fb1d05e6704 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:18.752597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 5/9] 10.222.0.61 () {56 vars in 1198 bytes} [Thu Sep 29 01:58:18 2022] GET /identity/v3/users/4aa345955ed042db9837b36c3d6532b0 => generated 247 bytes in 22 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:21.000934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f517b4-d267-484d-84ce-4544fd319302 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.001533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f517b4-d267-484d-84ce-4544fd319302 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.001533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f517b4-d267-484d-84ce-4544fd319302 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.002801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 5/10] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:21.008290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf63e7-fb74-4f59-ad67-9be65e44756d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.008756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf63e7-fb74-4f59-ad67-9be65e44756d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.008844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbf63e7-fb74-4f59-ad67-9be65e44756d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.011446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef1d18-8d7c-4282-b8ce-ae68bc44d626 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.011728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef1d18-8d7c-4282-b8ce-ae68bc44d626 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.011989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ef1d18-8d7c-4282-b8ce-ae68bc44d626 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.013948 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 6/11] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:21.018775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.019056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.019326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.028186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8fbf63e7-fb74-4f59-ad67-9be65e44756d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:21.029362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8fbf63e7-fb74-4f59-ad67-9be65e44756d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:21.062826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG passlib.handlers.bcrypt [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81684) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 29 01:58:21.063219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG passlib.handlers.bcrypt [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81684) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 29 01:58:21.112779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:21.114429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:21.116308 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 6/12] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:21.215556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 7/13] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:21.376994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf63e7-fb74-4f59-ad67-9be65e44756d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.401766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.476820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.483992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbf63e7-fb74-4f59-ad67-9be65e44756d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.517119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.517764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.518048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.518325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.585225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f20aa2a-3439-4c84-bb6b-98172f6f1883 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.587896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.589225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.590768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.613677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.622558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.623670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.623949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.624214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.635703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:21.644225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:21.645227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:21.646479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 8/14] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:21.651465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.688878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 7/15] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:21.694508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.748838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7413aff8-b0ed-4817-ae6c-f20e1340b506 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.781626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.782259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.782532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.782798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.783475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:21.783780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.785221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.790126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-303ed8ff-0e93-4fd6-909a-fed98cc58903 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.796796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 9/16] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:21.802830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.813174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.826638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.827286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.827567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.827840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.829521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:21.838514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.839987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:21.858759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:21.859639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:21.860459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:21.860459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:21.860459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:58:21.860459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 8/17] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:21.865944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.904866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c178dee2-b91b-477e-b619-559601b15855 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.942605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:21.943584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:21.943870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:21.944138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] PATH_INFO: `/v3/users/4aa345955ed042db9837b36c3d6532b0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:21.946447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] RBAC: Authorizing `identity:get_user(user_id=4aa345955ed042db9837b36c3d6532b0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:21.948431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:21.950339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 10/18] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/users/4aa345955ed042db9837b36c3d6532b0 => generated 247 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:21.955535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:21.966117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-408be10a-e385-4be3-aeba-a62d1b18f9bc None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:21.999898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.000512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.000790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.001064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.001705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:22.002010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:22.003474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:22.021839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 9/19] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:22.052598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.062320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b5bd16-8163-4a9f-9436-b6267acd1af5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.101759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3342effa-8b97-436f-9739-a7bc78fb6114 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.102459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342effa-8b97-436f-9739-a7bc78fb6114 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.102734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342effa-8b97-436f-9739-a7bc78fb6114 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.103006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3342effa-8b97-436f-9739-a7bc78fb6114 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.104244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3342effa-8b97-436f-9739-a7bc78fb6114 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:22.105740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3342effa-8b97-436f-9739-a7bc78fb6114 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:22.107363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 11/20] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:22.113723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:22.150743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:22.222047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf0a6769-48bf-41fb-a0fc-327a52dfd08d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:22.255247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc4b8a48-1a95-41af-b463-32c04327f430 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:22.255847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4b8a48-1a95-41af-b463-32c04327f430 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:22.256126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4b8a48-1a95-41af-b463-32c04327f430 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:22.256386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc4b8a48-1a95-41af-b463-32c04327f430 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:22.257217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4b8a48-1a95-41af-b463-32c04327f430 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.domain.id=default, user.id=4aa345955ed042db9837b36c3d6532b0) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:22.257512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4b8a48-1a95-41af-b463-32c04327f430 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:22.259010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc4b8a48-1a95-41af-b463-32c04327f430 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:22.266639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 10/21] 10.222.0.61 () {56 vars in 1322 bytes} [Thu Sep 29 01:58:22 2022] GET /identity/v3/role_assignments?user.id=4aa345955ed042db9837b36c3d6532b0&scope.domain.id=default&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 233 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:24.632801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f33a7f8-1b9b-4f29-a471-0202740f625a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.633388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f33a7f8-1b9b-4f29-a471-0202740f625a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.633388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f33a7f8-1b9b-4f29-a471-0202740f625a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.634739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 12/22] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:24.640699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8843db71-2567-46ec-87ce-e1a6741cc251 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.641523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8843db71-2567-46ec-87ce-e1a6741cc251 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.642052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8843db71-2567-46ec-87ce-e1a6741cc251 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.666624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8843db71-2567-46ec-87ce-e1a6741cc251 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:24.668269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8843db71-2567-46ec-87ce-e1a6741cc251 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:24.675063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee7cd8d-1e30-4893-aece-70cb1f8edb91 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.675606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee7cd8d-1e30-4893-aece-70cb1f8edb91 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.676178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ee7cd8d-1e30-4893-aece-70cb1f8edb91 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.679707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 13/23] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:24.690722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840f9451-2bdd-45cd-a1d9-04fcad519e9e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.691288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840f9451-2bdd-45cd-a1d9-04fcad519e9e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.691827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-840f9451-2bdd-45cd-a1d9-04fcad519e9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.722117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-840f9451-2bdd-45cd-a1d9-04fcad519e9e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:24.724974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-840f9451-2bdd-45cd-a1d9-04fcad519e9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:24.818025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 11/24] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:24.818025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bf4b6e-7c8e-4d31-84ed-768107f7404a None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.818025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bf4b6e-7c8e-4d31-84ed-768107f7404a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.818025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29bf4b6e-7c8e-4d31-84ed-768107f7404a None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.818025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 12/25] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:24 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:24.829812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 14/26] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:24.831064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15d7c7a-3e00-4440-9e95-c90497071553 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:24.831336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15d7c7a-3e00-4440-9e95-c90497071553 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:24.831611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b15d7c7a-3e00-4440-9e95-c90497071553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:24.869770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b15d7c7a-3e00-4440-9e95-c90497071553 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:24.870976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b15d7c7a-3e00-4440-9e95-c90497071553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:24.912770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:24.969196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 13/27] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:25.386224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840f9451-2bdd-45cd-a1d9-04fcad519e9e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:25.399686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15d7c7a-3e00-4440-9e95-c90497071553 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:25.513659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:25.527332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-840f9451-2bdd-45cd-a1d9-04fcad519e9e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:25.527538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b15d7c7a-3e00-4440-9e95-c90497071553 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:25.571509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:25.572492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:25.572964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:25.573427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:25.573597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f827c6d3-dc7a-4829-b473-92d18d06077b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:25.574422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:25.574535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827c6d3-dc7a-4829-b473-92d18d06077b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:25.574723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827c6d3-dc7a-4829-b473-92d18d06077b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:25.574998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f827c6d3-dc7a-4829-b473-92d18d06077b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:25.576491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:25.597707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f827c6d3-dc7a-4829-b473-92d18d06077b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:25.599878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f827c6d3-dc7a-4829-b473-92d18d06077b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f827c6d3-dc7a-4829-b473-92d18d06077b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:58:25.616741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:25.617664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:58:25.618499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:58:25.618499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:58:25.618499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:25.620274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 15/28] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:25.622710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c43faf8-cd67-4978-9cb7-f2cf0e63f009 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:25.626862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c43faf8-cd67-4978-9cb7-f2cf0e63f009 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:25.627435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c43faf8-cd67-4978-9cb7-f2cf0e63f009 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:25.628917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 16/29] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:25 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:25.630128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c43faf8-cd67-4978-9cb7-f2cf0e63f009 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:25.728572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 14/30] 10.222.0.61 () {60 vars in 1131 bytes} [Thu Sep 29 01:58:25 2022] POST /identity/v3/roles => generated 219 bytes in 329 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:58:25.736007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:25.740706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c43faf8-cd67-4978-9cb7-f2cf0e63f009 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:25.788931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:25.824098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa658c81-93a7-4e41-8f9b-41e3a47ce0d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:25.824881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa658c81-93a7-4e41-8f9b-41e3a47ce0d5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:25.825057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa658c81-93a7-4e41-8f9b-41e3a47ce0d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:25.825297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa658c81-93a7-4e41-8f9b-41e3a47ce0d5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:25.843074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa658c81-93a7-4e41-8f9b-41e3a47ce0d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-fa658c81-93a7-4e41-8f9b-41e3a47ce0d5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:25.849361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:25.850465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:25.851435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:25.853534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 17/31] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:25 2022] GET /identity/v3/roles/admin => generated 83 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:25.855254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e07950-192a-475a-be21-5961ba46c86b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:25.857328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e07950-192a-475a-be21-5961ba46c86b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:25.857739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e07950-192a-475a-be21-5961ba46c86b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:25.879058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:25.895448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b2e07950-192a-475a-be21-5961ba46c86b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:25.896690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b2e07950-192a-475a-be21-5961ba46c86b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:25.961883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c63c92c6-5b22-4556-8d57-4cebdd9b1a8c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:26.000550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 18/32] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:26.001409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e07950-192a-475a-be21-5961ba46c86b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:26.021063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60551c7d-003c-4fc5-9455-2636955f4b85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:26.021698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60551c7d-003c-4fc5-9455-2636955f4b85 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.021973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60551c7d-003c-4fc5-9455-2636955f4b85 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.022277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60551c7d-003c-4fc5-9455-2636955f4b85 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.023032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60551c7d-003c-4fc5-9455-2636955f4b85 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:26.023334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60551c7d-003c-4fc5-9455-2636955f4b85 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:26.024888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60551c7d-003c-4fc5-9455-2636955f4b85 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:26.043189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 15/33] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:58:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:26.044551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af22f09-f2fe-4e23-84c7-80ee2c17cbfe None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.045081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af22f09-f2fe-4e23-84c7-80ee2c17cbfe None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.045403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af22f09-f2fe-4e23-84c7-80ee2c17cbfe None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.047723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 16/34] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:26.052916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af22f09-f2fe-4e23-84c7-80ee2c17cbfe None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:26.130655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e07950-192a-475a-be21-5961ba46c86b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:26.154097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:26.170825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd9e3ca-1294-4ff4-8d1d-a3e3fabd7657 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:26.171558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af22f09-f2fe-4e23-84c7-80ee2c17cbfe None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:26.171767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd9e3ca-1294-4ff4-8d1d-a3e3fabd7657 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.172227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd9e3ca-1294-4ff4-8d1d-a3e3fabd7657 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.172695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd9e3ca-1294-4ff4-8d1d-a3e3fabd7657 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.173862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd9e3ca-1294-4ff4-8d1d-a3e3fabd7657 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:26.174409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd9e3ca-1294-4ff4-8d1d-a3e3fabd7657 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:26.176906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd9e3ca-1294-4ff4-8d1d-a3e3fabd7657 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:26.178726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:26.187764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 19/35] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:26.188768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42311a6f-8e4c-4b2e-ac6b-11915427767c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.189048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42311a6f-8e4c-4b2e-ac6b-11915427767c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.189319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42311a6f-8e4c-4b2e-ac6b-11915427767c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.210537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-42311a6f-8e4c-4b2e-ac6b-11915427767c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:26.211604 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-42311a6f-8e4c-4b2e-ac6b-11915427767c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:26.213288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:26.214334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.214866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.215385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.216237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:26.218080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:26.253568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 17/36] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 01:58:26 2022] POST /identity/v3/projects => generated 301 bytes in 202 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:26.254858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:26.323451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 20/37] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:26.341764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42311a6f-8e4c-4b2e-ac6b-11915427767c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:26.436471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f04e7f45-3bd2-4408-b81f-862e9d59957d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:26.462616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42311a6f-8e4c-4b2e-ac6b-11915427767c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:26.468808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:26.480348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93c784bc-52fe-42e4-9d3b-6780e1e62ee4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:26.481001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c784bc-52fe-42e4-9d3b-6780e1e62ee4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.481282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c784bc-52fe-42e4-9d3b-6780e1e62ee4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.481550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93c784bc-52fe-42e4-9d3b-6780e1e62ee4 None admin] PATH_INFO: `/v3/users/4aa345955ed042db9837b36c3d6532b0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.482735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c784bc-52fe-42e4-9d3b-6780e1e62ee4 None admin] RBAC: Authorizing `identity:get_user(user_id=4aa345955ed042db9837b36c3d6532b0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:26.484228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93c784bc-52fe-42e4-9d3b-6780e1e62ee4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:26.490156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 18/38] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:26 2022] GET /identity/v3/users/4aa345955ed042db9837b36c3d6532b0 => generated 247 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:26.491863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac67606-a43e-4576-878b-9d6c6cff925f None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.492150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac67606-a43e-4576-878b-9d6c6cff925f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.492440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac67606-a43e-4576-878b-9d6c6cff925f None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.494411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 19/39] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:26.495085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac67606-a43e-4576-878b-9d6c6cff925f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:26.504454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:26.516381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:26.517611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.518068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.518493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.519390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:26.521595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:26.613229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac67606-a43e-4576-878b-9d6c6cff925f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:26.645821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42606a2f-c2fd-4e8d-8bf0-40010d124716 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:26.646478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42606a2f-c2fd-4e8d-8bf0-40010d124716 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.646752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42606a2f-c2fd-4e8d-8bf0-40010d124716 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.647021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42606a2f-c2fd-4e8d-8bf0-40010d124716 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.648143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42606a2f-c2fd-4e8d-8bf0-40010d124716 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:26.649603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42606a2f-c2fd-4e8d-8bf0-40010d124716 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:26.651198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 20/40] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:26.652274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fbba18-f8bf-4b4f-b346-55be14497d08 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.652550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fbba18-f8bf-4b4f-b346-55be14497d08 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.652874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07fbba18-f8bf-4b4f-b346-55be14497d08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.663040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 21/41] 10.222.0.61 () {60 vars in 1131 bytes} [Thu Sep 29 01:58:26 2022] POST /identity/v3/roles => generated 225 bytes in 322 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:26.664387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:26.671791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-07fbba18-f8bf-4b4f-b346-55be14497d08 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:26.673368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-07fbba18-f8bf-4b4f-b346-55be14497d08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:26.740412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:26.749211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 21/42] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:26.750098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fbba18-f8bf-4b4f-b346-55be14497d08 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:26.809675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-090a00c9-a566-4f28-865a-7f384f0ce159 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:26.853707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:26.854361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:26.854636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:26.854902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:26.856371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:26.857891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:26.876799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 22/43] 10.222.0.61 () {60 vars in 1131 bytes} [Thu Sep 29 01:58:26 2022] POST /identity/v3/roles => generated 223 bytes in 213 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:26.878724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07fbba18-f8bf-4b4f-b346-55be14497d08 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:27.034037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:27.035021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.035431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.035834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] PATH_INFO: `/v3/domains/default/users/4aa345955ed042db9837b36c3d6532b0/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.037965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=4aa345955ed042db9837b36c3d6532b0, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:27.039827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:27.056338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:27.057933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 22/44] 10.222.0.61 () {58 vars in 1361 bytes} [Thu Sep 29 01:58:26 2022] PUT /identity/v3/domains/default/users/4aa345955ed042db9837b36c3d6532b0/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:27.115109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:27.180147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bc63dae-205b-49da-9446-412ec0ba9cb1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:27.215271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5191b5-d7d4-4dc9-8049-30dd7136e41a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:27.215918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5191b5-d7d4-4dc9-8049-30dd7136e41a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.216194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5191b5-d7d4-4dc9-8049-30dd7136e41a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.216462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5191b5-d7d4-4dc9-8049-30dd7136e41a None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.217753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5191b5-d7d4-4dc9-8049-30dd7136e41a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:27.219262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5191b5-d7d4-4dc9-8049-30dd7136e41a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:27.220870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 23/45] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:27 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:27.227177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:27.290179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:27.335608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec3507c-50d5-4555-a615-2a701db83448 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:27.370395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a940df3c-59db-4e41-a8e9-f5c2bb84ee87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:27.370926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a940df3c-59db-4e41-a8e9-f5c2bb84ee87 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:27.371206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a940df3c-59db-4e41-a8e9-f5c2bb84ee87 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:27.371480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a940df3c-59db-4e41-a8e9-f5c2bb84ee87 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:27.372136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a940df3c-59db-4e41-a8e9-f5c2bb84ee87 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:27.373628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a940df3c-59db-4e41-a8e9-f5c2bb84ee87 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:27.401967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 23/46] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 01:58:27 2022] POST /identity/v3/projects => generated 312 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:29.203338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870c6a7-2a21-4023-8ab3-7a7ea77c34a0 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.203925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870c6a7-2a21-4023-8ab3-7a7ea77c34a0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.203925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f870c6a7-2a21-4023-8ab3-7a7ea77c34a0 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.205226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 24/47] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:29.210713 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46f90b-2dc8-4fc0-8acf-c8a6cfe9615e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.211150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46f90b-2dc8-4fc0-8acf-c8a6cfe9615e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.211405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d46f90b-2dc8-4fc0-8acf-c8a6cfe9615e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.230863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5d46f90b-2dc8-4fc0-8acf-c8a6cfe9615e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:29.232060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5d46f90b-2dc8-4fc0-8acf-c8a6cfe9615e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:29.304881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 24/48] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:29.524879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9acf5-bcee-4807-aaa8-e93a2c3101e8 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.525061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9acf5-bcee-4807-aaa8-e93a2c3101e8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.525224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32e9acf5-bcee-4807-aaa8-e93a2c3101e8 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.526613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 25/49] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:29.532145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9054e-f33a-4d88-9774-8605658a3b69 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.532452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9054e-f33a-4d88-9774-8605658a3b69 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.532711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae9054e-f33a-4d88-9774-8605658a3b69 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.556122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9ae9054e-f33a-4d88-9774-8605658a3b69 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:29.557322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9ae9054e-f33a-4d88-9774-8605658a3b69 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:29.572038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e9acf5-bcee-4807-aaa8-e93a2c3101e8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:29.588407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:29.628628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 25/50] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:29.674600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32e9acf5-bcee-4807-aaa8-e93a2c3101e8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:29.688386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:29.709387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:29.710037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.710325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.710945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.721935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:29.727021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:29.727879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:29.728627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 26/51] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:29 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:29.734195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae9054e-f33a-4d88-9774-8605658a3b69 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:29.836874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae9054e-f33a-4d88-9774-8605658a3b69 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:29.871156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:29.871763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:29.872045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:29.872320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:29.872988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:29.873293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:29.874741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:29.886861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 26/52] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:29 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:29.891537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:29.892622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:29.924653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:29.992661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf6a956-0751-4b1e-b26d-bf079e21418d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:29.996245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfcfb58-4bd5-4217-95b6-150cf1d3faa2 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:30.025350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:30.025985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:30.026548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:30.026548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:30.027671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:30.029143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:30.029344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:30.029976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:30.030280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:30.030546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] PATH_INFO: `/v3/users/4aa345955ed042db9837b36c3d6532b0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:30.030733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 27/53] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:30.031978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] RBAC: Authorizing `identity:get_user(user_id=4aa345955ed042db9837b36c3d6532b0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:30.033415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:30.035000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 27/54] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:29 2022] GET /identity/v3/users/4aa345955ed042db9837b36c3d6532b0 => generated 247 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:30.036291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:30.040302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:30.079350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:30.133326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:30.137268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fe6f95-cd63-43cc-a9b9-27bb366d963a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:30.137398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3de5ca7f-d648-4b78-b685-b633b1408daa None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:30.171674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde6d03a-faa6-4c45-888b-521d9090b0db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:30.172287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6d03a-faa6-4c45-888b-521d9090b0db None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:30.172567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6d03a-faa6-4c45-888b-521d9090b0db None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:30.172838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde6d03a-faa6-4c45-888b-521d9090b0db None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:30.173075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:30.173494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde6d03a-faa6-4c45-888b-521d9090b0db None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:30.173710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:30.173987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:30.174292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:30.174978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde6d03a-faa6-4c45-888b-521d9090b0db None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:30.175407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:30.176856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:30.178827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 28/55] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:30 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:30.184601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:30.206373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 28/56] 10.222.0.61 () {60 vars in 1140 bytes} [Thu Sep 29 01:58:30 2022] POST /identity/v3/projects => generated 298 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:30.303461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97eb666f-3286-47ca-99ab-69bd4c186875 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:30.342590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8e14d5-c1e0-4411-9112-be416f0848f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:30.343216 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e14d5-c1e0-4411-9112-be416f0848f2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:30.343499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e14d5-c1e0-4411-9112-be416f0848f2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:30.343774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8e14d5-c1e0-4411-9112-be416f0848f2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:30.344529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8e14d5-c1e0-4411-9112-be416f0848f2 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.domain.id=default, user.id=4aa345955ed042db9837b36c3d6532b0) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:30.344838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8e14d5-c1e0-4411-9112-be416f0848f2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:30.346373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c8e14d5-c1e0-4411-9112-be416f0848f2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:30.353314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 29/57] 10.222.0.61 () {56 vars in 1322 bytes} [Thu Sep 29 01:58:30 2022] GET /identity/v3/role_assignments?user.id=4aa345955ed042db9837b36c3d6532b0&scope.domain.id=default&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 530 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:32.350901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39198a2-9157-493e-9dde-02d4934e9070 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.351593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39198a2-9157-493e-9dde-02d4934e9070 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.351593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39198a2-9157-493e-9dde-02d4934e9070 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.352915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 29/58] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:32.359408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a0be06-305d-4b95-8aed-7a510d490b04 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.360096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a0be06-305d-4b95-8aed-7a510d490b04 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.360239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a0be06-305d-4b95-8aed-7a510d490b04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.382665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-99a0be06-305d-4b95-8aed-7a510d490b04 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:32.384211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-99a0be06-305d-4b95-8aed-7a510d490b04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:32.408259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.455397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 30/59] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:32.728146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39198a2-9157-493e-9dde-02d4934e9070 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.830236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:32.839542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39198a2-9157-493e-9dde-02d4934e9070 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:32.874997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc82e5cd-62be-4b90-ba7e-5f24bf357964 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:32.875617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc82e5cd-62be-4b90-ba7e-5f24bf357964 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:32.875890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc82e5cd-62be-4b90-ba7e-5f24bf357964 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:32.876159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc82e5cd-62be-4b90-ba7e-5f24bf357964 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:32.877403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc82e5cd-62be-4b90-ba7e-5f24bf357964 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:32.878903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc82e5cd-62be-4b90-ba7e-5f24bf357964 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:32.880455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 30/60] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:32 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:32.886922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a0be06-305d-4b95-8aed-7a510d490b04 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:32.999851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a0be06-305d-4b95-8aed-7a510d490b04 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:33.035823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fe7297-f0a3-4e06-9e62-c787404ce42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:33.036423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fe7297-f0a3-4e06-9e62-c787404ce42e None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:33.036694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fe7297-f0a3-4e06-9e62-c787404ce42e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:33.036958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fe7297-f0a3-4e06-9e62-c787404ce42e None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:33.037598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fe7297-f0a3-4e06-9e62-c787404ce42e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:33.039137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fe7297-f0a3-4e06-9e62-c787404ce42e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:33.073221 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 31/61] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:32 2022] POST /identity/v3/users => generated 275 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:36.346990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a895d0c7-1eba-4071-a658-1c35620863cf None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.348051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a895d0c7-1eba-4071-a658-1c35620863cf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.348547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a895d0c7-1eba-4071-a658-1c35620863cf None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.350416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 31/62] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:36.356211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d8037a-f349-4148-b524-e54782eca04f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.357052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d8037a-f349-4148-b524-e54782eca04f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.357533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d8037a-f349-4148-b524-e54782eca04f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.383566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-e0d8037a-f349-4148-b524-e54782eca04f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:36.385106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e0d8037a-f349-4148-b524-e54782eca04f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:36.398604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:36.478346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 32/63] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:36.526071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc3803-4a74-4213-a89c-de3073600240 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.526736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc3803-4a74-4213-a89c-de3073600240 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.527290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cc3803-4a74-4213-a89c-de3073600240 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.528898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 32/64] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:36.534857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c923d250-22b4-4508-aaf9-1a6ee19d5f0c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.535140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c923d250-22b4-4508-aaf9-1a6ee19d5f0c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.535418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c923d250-22b4-4508-aaf9-1a6ee19d5f0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.563146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c923d250-22b4-4508-aaf9-1a6ee19d5f0c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:36.564565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c923d250-22b4-4508-aaf9-1a6ee19d5f0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:36.667625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 33/65] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:36.742767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef15099-bf9c-4cbd-b367-fab53abf9363 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.743438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef15099-bf9c-4cbd-b367-fab53abf9363 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.743928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ef15099-bf9c-4cbd-b367-fab53abf9363 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.745502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 33/66] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:36.751194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1e157d-d718-450b-841a-9e58502257e6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.751483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1e157d-d718-450b-841a-9e58502257e6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.751761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1e157d-d718-450b-841a-9e58502257e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:36.770756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7b1e157d-d718-450b-841a-9e58502257e6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:36.771959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7b1e157d-d718-450b-841a-9e58502257e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:36.774450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef15099-bf9c-4cbd-b367-fab53abf9363 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:36.882470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 34/67] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:36.939629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ef15099-bf9c-4cbd-b367-fab53abf9363 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:36.996333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:36.997405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:36.997885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:36.998452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.005888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1e157d-d718-450b-841a-9e58502257e6 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.016674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:37.021917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:37.022794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:37.023791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:37.026043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.027049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 34/68] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:36 2022] GET /identity/v3/roles/member => generated 84 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:37.036545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.155392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1e157d-d718-450b-841a-9e58502257e6 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.183562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.192989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e60d223-67b1-42b3-8427-f6b6539e77d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.193570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60d223-67b1-42b3-8427-f6b6539e77d5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.193849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60d223-67b1-42b3-8427-f6b6539e77d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.194140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e60d223-67b1-42b3-8427-f6b6539e77d5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.207665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e60d223-67b1-42b3-8427-f6b6539e77d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-7e60d223-67b1-42b3-8427-f6b6539e77d5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:37.214794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:37.215637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:58:37.216376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 35/69] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles/admin => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:37.217595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3332a8-8707-4c46-bb37-cd9d7855fa86 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.217595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3332a8-8707-4c46-bb37-cd9d7855fa86 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.217595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3332a8-8707-4c46-bb37-cd9d7855fa86 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.218967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 36/70] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:37.219842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428ca72-af31-4807-8479-1b8a9bd1562d None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.220116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428ca72-af31-4807-8479-1b8a9bd1562d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.220382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2428ca72-af31-4807-8479-1b8a9bd1562d None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.221881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 37/71] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:37 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:37.226807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428ca72-af31-4807-8479-1b8a9bd1562d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.272123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88da0be6-b0e3-4b93-bee8-26bfc1b26067 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.294641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.306849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2552d34d-598f-46f9-919a-b497b504c0c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.307448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552d34d-598f-46f9-919a-b497b504c0c5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.307725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552d34d-598f-46f9-919a-b497b504c0c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.307999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2552d34d-598f-46f9-919a-b497b504c0c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.308655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2552d34d-598f-46f9-919a-b497b504c0c5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:37.308959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2552d34d-598f-46f9-919a-b497b504c0c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:37.310375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2552d34d-598f-46f9-919a-b497b504c0c5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:37.323261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 35/72] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:37.323261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aace6cf-6228-4d26-82be-7d4775b9e3f3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.323261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aace6cf-6228-4d26-82be-7d4775b9e3f3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.323261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aace6cf-6228-4d26-82be-7d4775b9e3f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.330944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2428ca72-af31-4807-8479-1b8a9bd1562d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.339173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0aace6cf-6228-4d26-82be-7d4775b9e3f3 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:37.340687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0aace6cf-6228-4d26-82be-7d4775b9e3f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:37.364948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58316ccf-da87-43c2-b58c-27875a3f68e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.365566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58316ccf-da87-43c2-b58c-27875a3f68e6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.365846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58316ccf-da87-43c2-b58c-27875a3f68e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.366145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58316ccf-da87-43c2-b58c-27875a3f68e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.366904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58316ccf-da87-43c2-b58c-27875a3f68e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:37.367256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58316ccf-da87-43c2-b58c-27875a3f68e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:37.368659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58316ccf-da87-43c2-b58c-27875a3f68e6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:37.377139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 38/73] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:37.383473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2071273f-808d-401e-82a3-2ed3cb94b96b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.383742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2071273f-808d-401e-82a3-2ed3cb94b96b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.384019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2071273f-808d-401e-82a3-2ed3cb94b96b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.401408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2071273f-808d-401e-82a3-2ed3cb94b96b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:37.402658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2071273f-808d-401e-82a3-2ed3cb94b96b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:37.407472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 36/74] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:37.410850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aace6cf-6228-4d26-82be-7d4775b9e3f3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.426736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.475027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 39/75] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:37.475883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2071273f-808d-401e-82a3-2ed3cb94b96b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.528301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aace6cf-6228-4d26-82be-7d4775b9e3f3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.552494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.573819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.574476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.574747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.575015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.587651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:37.592544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:37.593401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 37/76] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:37.598691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.610874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2071273f-808d-401e-82a3-2ed3cb94b96b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.645837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.646723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.647160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.647560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.661583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:37.663344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:37.665237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 40/77] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:37.666330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.695731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.722330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e4296e-2db7-4803-988a-1cc0509cd0f0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.758268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.758825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.759097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.759364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] PATH_INFO: `/v3/users/4aa345955ed042db9837b36c3d6532b0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.760499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] RBAC: Authorizing `identity:get_user(user_id=4aa345955ed042db9837b36c3d6532b0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:37.761947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:37.763679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 38/78] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/users/4aa345955ed042db9837b36c3d6532b0 => generated 247 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:37.764365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.773270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56537686-d4fb-46ad-aae4-67efc8f9a0ab None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.789571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.807417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.808033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.808311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.808581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.809244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:37.809544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:37.810956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:37.818871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 41/79] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:37.819641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.862371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95da4b59-329b-44a0-b9bb-7fc0344d2a72 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.896458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.897172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.897515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.897846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:37.899290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:37.900724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:37.902320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 39/80] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:37.903239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:37.917751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:37.923939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df02a13-fe28-4443-ad24-c4889b9d99e2 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:37.957760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:37.958401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:37.958672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:37.958937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.002402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b321cf-5864-4c5a-97b8-2b21c71c4968 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.003369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.026579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.028067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.037658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.038774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.039993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.040482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.052464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:38.057545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:38.058562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 40/81] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:38.062952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.126817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.131916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 42/82] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:38.133053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.161260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdbea24-f564-4efd-a44c-03bf7857c0f1 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.195028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.196235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.196514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.196783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.212027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.234213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b4191f3-0058-454b-a9c8-7a25ff808bd2 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.264351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.265900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.267956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.268598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.268877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.269151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.270313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.271772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.273383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 43/83] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:38.274202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.332688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.364411 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 41/84] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:38.365678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.374048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb09c7d-e3fd-470c-ac6e-7dae7f2c0d56 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.408082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.408694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.408969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.409242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.409894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.411398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.434717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 44/85] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:38 2022] POST /identity/v3/users => generated 289 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:38.443138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.446369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.475436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acd81e33-3e6f-4cd5-8689-2b2f018148dd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.515446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.516262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.516834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.516834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.517506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:38.517768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.519253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.530412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 42/86] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:38.531603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.569044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89c0572e-8bfb-4b7f-add9-ffdac8f5775e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.571689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.606787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.607512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.607795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.608230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.609343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.610871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.612762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 45/87] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:38.613469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.655939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b13393-5e55-4bb3-b6a6-bc05f2249627 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.698892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.699512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.699786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.700054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.708156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.751916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc692e8c-4008-4aee-8041-834b994ed1ba None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.796231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.796456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.796919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.797207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.797484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.797718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.798662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.801389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.803145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 46/88] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:38.804273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.835571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:38.920893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 43/89] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:38.921713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.922056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6280c1c8-a121-445b-82a0-202f2407c047 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:38.966898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:38.968419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:38.968703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:38.968975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:38.970164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:38.971627 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:38.973958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 47/90] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:38.974607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:38.986156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.038071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d45554f-7ae1-4388-81c6-b091b766d576 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.072816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.078043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.078956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.079515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.079962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.080755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=9564038bafec44429094752520f2166c) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:39.081132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.082680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.088316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c40e8e78-f1e5-4e9a-9bbc-992829ac89ee None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.092408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 44/91] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/role_assignments?user.id=9564038bafec44429094752520f2166c&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:39.093191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.126106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.126737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.127019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.127294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.128132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=4aa345955ed042db9837b36c3d6532b0) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:39.128453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.130054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.141533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 48/92] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:38 2022] GET /identity/v3/role_assignments?user.id=4aa345955ed042db9837b36c3d6532b0&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:39.142669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.249125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e74031d-3bf2-4c4d-8df9-6da43c45b8bb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.286392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.324143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.324803 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.325081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.325355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.331043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.332572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.334450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 45/93] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:39.341396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.357921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ff885b6-8f31-4838-ad9e-92f8fcc0e459 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.442530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.474816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.475526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.475742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.476015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.478389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ac1a968-121e-47c1-9c46-2eb938cf5e34 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.517849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.518473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.518753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.519071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.519810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] RBAC: Adding query filter params (role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=9564038bafec44429094752520f2166c) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:39.520121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.521636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.528367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 46/94] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/role_assignments?user.id=9564038bafec44429094752520f2166c&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:39.618656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.620514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.717418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.756784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 49/95] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 615 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:39.765837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:39.889251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bc2f4b2-fd8c-4677-81aa-b2f81f38f14f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:39.918783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:39.927040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ea0249-bfb3-4529-9877-582ea3d9cfc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:39.927856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ea0249-bfb3-4529-9877-582ea3d9cfc8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:39.928191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ea0249-bfb3-4529-9877-582ea3d9cfc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:39.929000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ea0249-bfb3-4529-9877-582ea3d9cfc8 None admin] PATH_INFO: `/v3/projects/d959227356ef497d882b4ac75d6c2557` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:39.930844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ea0249-bfb3-4529-9877-582ea3d9cfc8 None admin] RBAC: Authorizing `identity:get_project(project_id=d959227356ef497d882b4ac75d6c2557)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:39.932323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5ea0249-bfb3-4529-9877-582ea3d9cfc8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:39.934310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 47/96] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/projects/d959227356ef497d882b4ac75d6c2557 => generated 312 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:39.940193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:40.085049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee80f7c-d604-4451-b366-adb8330f6c53 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:40.136767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23cb9d9d-d36d-47a0-a829-345cd15fec79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:40.138041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cb9d9d-d36d-47a0-a829-345cd15fec79 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:40.138785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cb9d9d-d36d-47a0-a829-345cd15fec79 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:40.139335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23cb9d9d-d36d-47a0-a829-345cd15fec79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:40.140278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cb9d9d-d36d-47a0-a829-345cd15fec79 None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=d959227356ef497d882b4ac75d6c2557, user.id=9564038bafec44429094752520f2166c) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:40.141100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cb9d9d-d36d-47a0-a829-345cd15fec79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:40.143132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23cb9d9d-d36d-47a0-a829-345cd15fec79 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:40.154452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 50/97] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:39 2022] GET /identity/v3/role_assignments?user.id=9564038bafec44429094752520f2166c&scope.project.id=d959227356ef497d882b4ac75d6c2557&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 259 bytes in 215 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:41.513391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79df8bd-eee4-4c09-94ec-a036734a0c96 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.513913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79df8bd-eee4-4c09-94ec-a036734a0c96 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.513913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f79df8bd-eee4-4c09-94ec-a036734a0c96 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.515314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 48/98] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:41.531834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0181f2-681e-4345-8ee0-10c0942fbaac None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:41.531834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0181f2-681e-4345-8ee0-10c0942fbaac None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:41.532344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0181f2-681e-4345-8ee0-10c0942fbaac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:41.558408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5c0181f2-681e-4345-8ee0-10c0942fbaac None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:41.559607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5c0181f2-681e-4345-8ee0-10c0942fbaac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:41.598040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:41.642858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 51/99] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:41.954342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79df8bd-eee4-4c09-94ec-a036734a0c96 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.078057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f79df8bd-eee4-4c09-94ec-a036734a0c96 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.113860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f30a6e49-fbf4-4458-903e-7c3a33714455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.114532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30a6e49-fbf4-4458-903e-7c3a33714455 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.114812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30a6e49-fbf4-4458-903e-7c3a33714455 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.115083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f30a6e49-fbf4-4458-903e-7c3a33714455 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.126114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f30a6e49-fbf4-4458-903e-7c3a33714455 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f30a6e49-fbf4-4458-903e-7c3a33714455 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:42.130955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:42.131798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:42.132574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 49/100] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:41 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:42.143424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0181f2-681e-4345-8ee0-10c0942fbaac None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.160303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6da92c9-edba-437c-865b-b22ca3e3ad9b None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.160594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6da92c9-edba-437c-865b-b22ca3e3ad9b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.160867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6da92c9-edba-437c-865b-b22ca3e3ad9b None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.163789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 50/101] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:42.171672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749e69c-9f39-4562-9bdd-a9b49706594b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.171964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749e69c-9f39-4562-9bdd-a9b49706594b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.172245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e749e69c-9f39-4562-9bdd-a9b49706594b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.204791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e749e69c-9f39-4562-9bdd-a9b49706594b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:42.206038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e749e69c-9f39-4562-9bdd-a9b49706594b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:42.313733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.347990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 51/102] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:42.350785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599ff19-a281-4440-b2a0-f16167a25027 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.351067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599ff19-a281-4440-b2a0-f16167a25027 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.351339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4599ff19-a281-4440-b2a0-f16167a25027 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.360528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 52/103] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:42 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:42.362431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0181f2-681e-4345-8ee0-10c0942fbaac None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.389202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a423ba-79a8-4a3a-9cc2-9c600404ed55 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.389562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a423ba-79a8-4a3a-9cc2-9c600404ed55 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.389743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a423ba-79a8-4a3a-9cc2-9c600404ed55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.416881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e5a423ba-79a8-4a3a-9cc2-9c600404ed55 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:42.418104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e5a423ba-79a8-4a3a-9cc2-9c600404ed55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:42.438755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.439431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.439706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.439974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.440686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:42.440990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.442410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.459564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 52/104] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 317 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:42.467346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.509628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.567309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 53/105] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 29 01:58:42.618674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7dfadb-f727-42cb-8922-8ed27e5146a1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.627613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb84a1d-f945-47c1-815e-6361fba00c3f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.627895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb84a1d-f945-47c1-815e-6361fba00c3f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.628180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb84a1d-f945-47c1-815e-6361fba00c3f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.629528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 54/106] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:42.636843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e77ebe-6c7e-4933-b4f9-1b75acb0cb3d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.637127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e77ebe-6c7e-4933-b4f9-1b75acb0cb3d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.637408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e77ebe-6c7e-4933-b4f9-1b75acb0cb3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.660169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-99e77ebe-6c7e-4933-b4f9-1b75acb0cb3d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:42.662062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-99e77ebe-6c7e-4933-b4f9-1b75acb0cb3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:42.679320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.680003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.680562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.681213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] PATH_INFO: `/v3/users/925a76c63ad5415097d7368c883d5c62` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.709265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] RBAC: Authorizing `identity:get_user(user_id=925a76c63ad5415097d7368c883d5c62)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:42.710872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:42.713212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 53/107] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/users/925a76c63ad5415097d7368c883d5c62 => generated 289 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:42.720259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.790325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 55/108] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:42.817771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e77ebe-6c7e-4933-b4f9-1b75acb0cb3d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:42.865379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.899397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9b63df-e88b-46d9-ac79-aa6746abcc16 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:42.902405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:42.980132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:42.980670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:42.980953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:42.981227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:42.992144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e77ebe-6c7e-4933-b4f9-1b75acb0cb3d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.047600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19e55ef-d339-499e-8e13-956f01a44e3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.048314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19e55ef-d339-499e-8e13-956f01a44e3c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.048595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19e55ef-d339-499e-8e13-956f01a44e3c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.048871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19e55ef-d339-499e-8e13-956f01a44e3c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.062705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19e55ef-d339-499e-8e13-956f01a44e3c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a19e55ef-d339-499e-8e13-956f01a44e3c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:43.070175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:43.071110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:43.071868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 56/109] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/roles/member => generated 84 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:43.073842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b29cf73-89b8-4947-847f-defefba542ad None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.078238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b29cf73-89b8-4947-847f-defefba542ad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.078756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b29cf73-89b8-4947-847f-defefba542ad None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.082603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 57/110] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:43 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:43.087076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b29cf73-89b8-4947-847f-defefba542ad None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.088773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.090350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.183615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.203228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 54/111] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 483 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:43.204558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.220386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b29cf73-89b8-4947-847f-defefba542ad None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.228347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.254660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-943b4430-329b-4bcf-a1c8-9d3b8f126a96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.255222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943b4430-329b-4bcf-a1c8-9d3b8f126a96 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.255500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943b4430-329b-4bcf-a1c8-9d3b8f126a96 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.255772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943b4430-329b-4bcf-a1c8-9d3b8f126a96 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.257215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943b4430-329b-4bcf-a1c8-9d3b8f126a96 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.258824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-943b4430-329b-4bcf-a1c8-9d3b8f126a96 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.260568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 58/112] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:43.261524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1018d93-a894-435d-9a04-3b3b68699570 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.261804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1018d93-a894-435d-9a04-3b3b68699570 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.262143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1018d93-a894-435d-9a04-3b3b68699570 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.280290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f1018d93-a894-435d-9a04-3b3b68699570 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:43.281556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f1018d93-a894-435d-9a04-3b3b68699570 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:43.321253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3fd15b-a00d-4af5-9ac0-e93c7e155874 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.348732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 59/113] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:43.362209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1018d93-a894-435d-9a04-3b3b68699570 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.368906 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.369536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.369814 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.370118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.370799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:43.371099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.372492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.387542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 55/114] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:43.390876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.442967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.457325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1018d93-a894-435d-9a04-3b3b68699570 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.460502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.494904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06012c3-5bb2-4f94-8177-4997cd635179 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.495649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06012c3-5bb2-4f94-8177-4997cd635179 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.496218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06012c3-5bb2-4f94-8177-4997cd635179 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.496400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06012c3-5bb2-4f94-8177-4997cd635179 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.498769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f012deba-7f5b-4a85-9025-5d638becc633 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.507239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06012c3-5bb2-4f94-8177-4997cd635179 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e06012c3-5bb2-4f94-8177-4997cd635179 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:43.512964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:43.513833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:43.514928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 60/115] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:43.515639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc9eb58-fd18-4af6-811c-ade72a3368ff None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.516290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc9eb58-fd18-4af6-811c-ade72a3368ff None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.516290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc9eb58-fd18-4af6-811c-ade72a3368ff None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.521961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 61/116] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:43.521961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc9eb58-fd18-4af6-811c-ade72a3368ff None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.547339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.548224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.548666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.549614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.551004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.552569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.554767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 56/117] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:43.555477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.633988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc9eb58-fd18-4af6-811c-ade72a3368ff None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.666188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a9f832-0d12-4ae4-9131-1566d7910fdd None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.667390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.668158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.669182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.669460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.670153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.671596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.679716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.698709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 62/118] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 01:58:43 2022] POST /identity/v3/projects => generated 302 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:43.699542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.705760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3ca2b7-2186-489d-b9ed-6984a49e7ff1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.706475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ca2b7-2186-489d-b9ed-6984a49e7ff1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.706740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ca2b7-2186-489d-b9ed-6984a49e7ff1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.707013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3ca2b7-2186-489d-b9ed-6984a49e7ff1 None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.709035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ca2b7-2186-489d-b9ed-6984a49e7ff1 None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.710682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f3ca2b7-2186-489d-b9ed-6984a49e7ff1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.717411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 57/119] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:43.717411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60325708-fe82-423e-a129-40cefd7e5934 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.717411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60325708-fe82-423e-a129-40cefd7e5934 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.717411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60325708-fe82-423e-a129-40cefd7e5934 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.731259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.736435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-60325708-fe82-423e-a129-40cefd7e5934 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:43.737625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-60325708-fe82-423e-a129-40cefd7e5934 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:43.818780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 58/120] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:43.819558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60325708-fe82-423e-a129-40cefd7e5934 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.838657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703ee37b-288e-47c7-9e14-8ebf0a47beaf None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.886706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:43.887256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:43.887609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:43.887951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:43.888706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:43.889071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:43.890838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:43.901342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 63/121] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:43.902034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:43.960605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60325708-fe82-423e-a129-40cefd7e5934 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:43.979096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:43.999889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.006088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.007219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.007585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.007931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.008961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] RBAC: Adding query filter params (role.id=837dd36862a94b73aa5d4d06027541fc, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=925a76c63ad5415097d7368c883d5c62) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:44.009351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.011448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.019159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 59/122] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/role_assignments?user.id=925a76c63ad5415097d7368c883d5c62&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=837dd36862a94b73aa5d4d06027541fc => generated 259 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.020250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.028392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84bf71c-71bd-48de-8c6a-f005ce619931 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.099360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.099932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.100214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.100505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.112968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:44.118548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:44.119474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:44.120524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 64/123] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:44.121299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.185068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ec82a6-f2dd-4660-904b-a6fb8cab3da8 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.234027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.234027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.234027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.234027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.234425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.234425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.234894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a27c09c-30fd-4c3c-9c79-30bc30c77b44 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.235964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 60/124] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.238431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.284133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.285064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.285359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.285640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.286847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.288331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.290329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.290486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 65/125] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.291570 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.304756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.350414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a2930e-0425-4992-a29b-e57f6ac6bf94 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.387137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.387811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.388299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.388441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] PATH_INFO: `/v3/roles/member` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.399577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:44.405605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:44.406702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 61/126] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:44.407643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.411024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59549f8e-777d-403f-950d-6c16eaf12492 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.449088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.450094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.450515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.450951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.451760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:44.452254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.453937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.464220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 66/127] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.465344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.515887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48536333-52f4-41f1-ba80-6bc6479635ff None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.519291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.547735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.561688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.562722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.563175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.563716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/9564038bafec44429094752520f2166c/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.565854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] RBAC: Authorizing `identity:create_grant(project_id=507e44cb9594402a9fd0ac69d80cd885, user_id=9564038bafec44429094752520f2166c, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.567767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.587965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 62/128] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:44 2022] PUT /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/9564038bafec44429094752520f2166c/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:44.588668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b789ac4-0961-41cb-b746-6b7be5f87f96 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.589365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.659702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.660257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.660532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.661163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.664052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.665839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.667897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 67/129] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.668736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e705dbc-f0b4-4baa-8612-9e18c612b140 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.769335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.770488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.770967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.771468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.772373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:44.772952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.774960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.787270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 63/130] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.788504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.824691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:44.917394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e41412-1b57-402d-9a20-2e177f320518 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.927249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac59928-f029-410e-a074-e0d2852e5b7c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:44.965859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.966576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.966857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.967130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/9564038bafec44429094752520f2166c/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.971183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] RBAC: Authorizing `identity:create_grant(project_id=507e44cb9594402a9fd0ac69d80cd885, user_id=9564038bafec44429094752520f2166c, role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.972652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.981811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:44.982850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:44.983324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:44.983788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] PATH_INFO: `/v3/users/4aa345955ed042db9837b36c3d6532b0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:44.985312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] RBAC: Authorizing `identity:get_user(user_id=4aa345955ed042db9837b36c3d6532b0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:44.987629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:44.993707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 68/131] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/users/4aa345955ed042db9837b36c3d6532b0 => generated 247 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:44.996057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 64/132] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:44 2022] PUT /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/9564038bafec44429094752520f2166c/roles/1b55a134a7b24115b2bb8691600d2619 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:44.996669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:44.998866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.005741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.126871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.136574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd53fd4-08c9-46f9-9303-60b5043d7e6f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.190355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b85759-d689-4907-86b3-691501713a53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.191415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b85759-d689-4907-86b3-691501713a53 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.193516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b85759-d689-4907-86b3-691501713a53 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.194044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b85759-d689-4907-86b3-691501713a53 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.197235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b85759-d689-4907-86b3-691501713a53 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.201085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b85759-d689-4907-86b3-691501713a53 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.204416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 65/133] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:45.212225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b85759-d689-4907-86b3-691501713a53 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9281390a-c3b4-4f73-8823-5e68da32a0e9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.285918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.287038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.287521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.288009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.289582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.291377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.293229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 69/134] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:44 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:45.299804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.416994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.417755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b85759-d689-4907-86b3-691501713a53 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.468009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f87ab9a6-c6f4-4590-88e1-eb88484e6d33 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.484796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.485475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.485747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.486029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/4aa345955ed042db9837b36c3d6532b0/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.489403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] RBAC: Authorizing `identity:create_grant(project_id=507e44cb9594402a9fd0ac69d80cd885, user_id=4aa345955ed042db9837b36c3d6532b0, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.490878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.516046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 66/135] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:45 2022] PUT /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/4aa345955ed042db9837b36c3d6532b0/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:45.531682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a39b0b9f-cca7-43b7-ae67-9364fd186ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.532396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39b0b9f-cca7-43b7-ae67-9364fd186ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.532672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39b0b9f-cca7-43b7-ae67-9364fd186ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.532938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a39b0b9f-cca7-43b7-ae67-9364fd186ad4 None admin] PATH_INFO: `/v3/projects/d959227356ef497d882b4ac75d6c2557` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.537267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39b0b9f-cca7-43b7-ae67-9364fd186ad4 None admin] RBAC: Authorizing `identity:get_project(project_id=d959227356ef497d882b4ac75d6c2557)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.538937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a39b0b9f-cca7-43b7-ae67-9364fd186ad4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.541315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 70/136] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:45 2022] GET /identity/v3/projects/d959227356ef497d882b4ac75d6c2557 => generated 312 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:45.547961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:45.571737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:45.676647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae2fe19-007e-49d6-8ef3-ba96ece7daae None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:45.730813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4978f8-cb58-4e76-8c13-b9300a073ea3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:45.731418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4978f8-cb58-4e76-8c13-b9300a073ea3 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:45.731674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4978f8-cb58-4e76-8c13-b9300a073ea3 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:45.731944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4978f8-cb58-4e76-8c13-b9300a073ea3 None admin] PATH_INFO: `/v3/projects/d959227356ef497d882b4ac75d6c2557/users/9564038bafec44429094752520f2166c/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:45.733988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4978f8-cb58-4e76-8c13-b9300a073ea3 None admin] RBAC: Authorizing `identity:create_grant(project_id=d959227356ef497d882b4ac75d6c2557, user_id=9564038bafec44429094752520f2166c, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:45.735607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc4978f8-cb58-4e76-8c13-b9300a073ea3 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:45.766797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 67/137] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:45 2022] PUT /identity/v3/projects/d959227356ef497d882b4ac75d6c2557/users/9564038bafec44429094752520f2166c/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:46.412457 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0d9cf-3c17-454a-9c81-421fedb7cc03 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.412975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0d9cf-3c17-454a-9c81-421fedb7cc03 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.412975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa0d9cf-3c17-454a-9c81-421fedb7cc03 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.416101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 71/138] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:46.421631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e39464c-b041-4ae1-b0fe-7cad708f52ae None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.421970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e39464c-b041-4ae1-b0fe-7cad708f52ae None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.422274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e39464c-b041-4ae1-b0fe-7cad708f52ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.451353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7e39464c-b041-4ae1-b0fe-7cad708f52ae None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:46.452577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7e39464c-b041-4ae1-b0fe-7cad708f52ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:46.533374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 68/139] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:46.666395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b04fd2-bcd2-4cb7-aa28-650158395036 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.666695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b04fd2-bcd2-4cb7-aa28-650158395036 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.666871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b04fd2-bcd2-4cb7-aa28-650158395036 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.668203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 72/140] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:46.673545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97af04-ecaf-49e5-8265-779e6010ecdc None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:46.673826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97af04-ecaf-49e5-8265-779e6010ecdc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:46.674121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b97af04-ecaf-49e5-8265-779e6010ecdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:46.684814 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:46.698396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5b97af04-ecaf-49e5-8265-779e6010ecdc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:46.699612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5b97af04-ecaf-49e5-8265-779e6010ecdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:46.780249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 69/141] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:46.959231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b04fd2-bcd2-4cb7-aa28-650158395036 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.061659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b04fd2-bcd2-4cb7-aa28-650158395036 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.095177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.095866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.096204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.096478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.097633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.099133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.100868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 73/142] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:47.106549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b97af04-ecaf-49e5-8265-779e6010ecdc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.216141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.227671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.231070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b97af04-ecaf-49e5-8265-779e6010ecdc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.291426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.292049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.292322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.292583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.293252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.294740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.311920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.351059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 70/143] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:47 2022] POST /identity/v3/users => generated 283 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:47.392674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6091c598-9a19-4a16-a382-0713d72e7134 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.441803 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.442623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.442812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.443086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.455385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:47.460733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:47.461684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:47.462496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:47.464904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 74/144] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/roles/reader => generated 84 bytes in 249 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:47.468922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.588536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b884520-9f6a-4ff9-9d0c-09dd4bbdcf4b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.635292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.635932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.636201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.636519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.637209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:47.637504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.638992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.653458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 71/145] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:47.659366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.744297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.781558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db1ab54-c97c-4e7c-a5df-9501dd3e4a65 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.817649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.818320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.818601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.818877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] PATH_INFO: `/v3/users/925a76c63ad5415097d7368c883d5c62` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.820014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] RBAC: Authorizing `identity:get_user(user_id=925a76c63ad5415097d7368c883d5c62)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.821551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.823195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 75/146] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/users/925a76c63ad5415097d7368c883d5c62 => generated 289 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:47.828850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:47.888157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:INSERT=3,keystone:SELECT=97 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:47.943165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0e33e99-357c-4fe5-af7b-f7b6857f8198 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:47.992307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4ef937-f09b-4671-bab5-da299cc5fac2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:47.993071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ef937-f09b-4671-bab5-da299cc5fac2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:47.993354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ef937-f09b-4671-bab5-da299cc5fac2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:47.993629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ef937-f09b-4671-bab5-da299cc5fac2 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:47.994979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4ef937-f09b-4671-bab5-da299cc5fac2 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:47.996434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4ef937-f09b-4671-bab5-da299cc5fac2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:47.998581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 72/147] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:47 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:48.004417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.130966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f02b1c4-2dc9-4c48-9466-c17b4e22c516 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.174262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a091b76b-4f89-4e03-9554-7a7fdcf57bc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.174949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a091b76b-4f89-4e03-9554-7a7fdcf57bc6 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.175228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a091b76b-4f89-4e03-9554-7a7fdcf57bc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.175501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a091b76b-4f89-4e03-9554-7a7fdcf57bc6 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/925a76c63ad5415097d7368c883d5c62/roles/837dd36862a94b73aa5d4d06027541fc` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.177419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a091b76b-4f89-4e03-9554-7a7fdcf57bc6 None admin] RBAC: Authorizing `identity:create_grant(project_id=507e44cb9594402a9fd0ac69d80cd885, user_id=925a76c63ad5415097d7368c883d5c62, role_id=837dd36862a94b73aa5d4d06027541fc)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:48.178917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a091b76b-4f89-4e03-9554-7a7fdcf57bc6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:48.183555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbefb803-4ead-4ff5-8614-1fc758e67244 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.183838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbefb803-4ead-4ff5-8614-1fc758e67244 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.184112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbefb803-4ead-4ff5-8614-1fc758e67244 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.184479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.185523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 73/148] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:48 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:48.192378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6ecd94-d486-4593-8e44-c9003c085569 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.192669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6ecd94-d486-4593-8e44-c9003c085569 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.193028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6ecd94-d486-4593-8e44-c9003c085569 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.206758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 76/149] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:48 2022] PUT /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885/users/925a76c63ad5415097d7368c883d5c62/roles/837dd36862a94b73aa5d4d06027541fc => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:48.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-af6ecd94-d486-4593-8e44-c9003c085569 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:48.224436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-af6ecd94-d486-4593-8e44-c9003c085569 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:48.288417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f114f36a-7f1f-4c3f-8cd1-18ac20aec63d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.288706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f114f36a-7f1f-4c3f-8cd1-18ac20aec63d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.288880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f114f36a-7f1f-4c3f-8cd1-18ac20aec63d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.290475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 77/150] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:58:48.296191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0e05ce-3a68-4012-8a4a-421fca583fa6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.296473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0e05ce-3a68-4012-8a4a-421fca583fa6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.296749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a0e05ce-3a68-4012-8a4a-421fca583fa6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.327082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 74/151] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 136 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:48.339883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4a0e05ce-3a68-4012-8a4a-421fca583fa6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:48.341514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4a0e05ce-3a68-4012-8a4a-421fca583fa6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:48.428363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 78/152] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:58:48.666949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6ecd94-d486-4593-8e44-c9003c085569 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.738730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:48.803094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0e05ce-3a68-4012-8a4a-421fca583fa6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.833366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6ecd94-d486-4593-8e44-c9003c085569 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:48.912225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39d8ba12-eb0c-4c53-8487-2e8befa9be09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:48.912225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d8ba12-eb0c-4c53-8487-2e8befa9be09 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.912632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d8ba12-eb0c-4c53-8487-2e8befa9be09 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.912806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39d8ba12-eb0c-4c53-8487-2e8befa9be09 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.941988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39d8ba12-eb0c-4c53-8487-2e8befa9be09 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-39d8ba12-eb0c-4c53-8487-2e8befa9be09 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:48.948781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:48.949764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:48.950568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:58:48.953654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 75/153] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 287 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:48.956839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3e10b1-e045-4bbd-baa3-e528846c8ec9 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:48.958168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3e10b1-e045-4bbd-baa3-e528846c8ec9 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:48.959441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac3e10b1-e045-4bbd-baa3-e528846c8ec9 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:48.963243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 76/154] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:48 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:48.965053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3e10b1-e045-4bbd-baa3-e528846c8ec9 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:48.992269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a0e05ce-3a68-4012-8a4a-421fca583fa6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.046355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.070951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540df956-3613-4b68-adb0-c8bf2b38522a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.071977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540df956-3613-4b68-adb0-c8bf2b38522a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.072262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540df956-3613-4b68-adb0-c8bf2b38522a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.072527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540df956-3613-4b68-adb0-c8bf2b38522a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.094798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540df956-3613-4b68-adb0-c8bf2b38522a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-540df956-3613-4b68-adb0-c8bf2b38522a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:49.105074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:49.105947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:49.106883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:49.108613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 79/155] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 306 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:49.109788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b74617-60b7-470f-9cb9-9bffd75f1467 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.110088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b74617-60b7-470f-9cb9-9bffd75f1467 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.110358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b74617-60b7-470f-9cb9-9bffd75f1467 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.145194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-73b74617-60b7-470f-9cb9-9bffd75f1467 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:49.146784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-73b74617-60b7-470f-9cb9-9bffd75f1467 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:49.155989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac3e10b1-e045-4bbd-baa3-e528846c8ec9 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.170217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.204068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.204681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.204950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.205213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.205878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:49.210079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.210079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.216658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 77/156] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:48 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:49.217503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.241488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 80/157] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:49.242730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b74617-60b7-470f-9cb9-9bffd75f1467 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.375760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42e4ad77-4274-4aea-b6df-82585f734035 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.429775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.430605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.430882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.431149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.431819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:49.432113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.436981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b74617-60b7-470f-9cb9-9bffd75f1467 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.439088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.448941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 78/158] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:49.456881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.467510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.510779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.511441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.511718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.511992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.513286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.518022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.518022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 81/159] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:49.521726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.522141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.629733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ffa72a6-cbd1-4e17-9d54-31469502151e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.683631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.685429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.688164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.688641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.690946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.692742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.695552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 79/160] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 239 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:49.702226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:49.760397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c5bb48-73b6-4f02-bf9c-a6188b85b869 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.829281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.833202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.834129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.835049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:49.893545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fb1394-7d9c-490c-92b7-56f0561e7e72 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:49.895069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.918499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:49.937208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:49.939124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:49.955124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:49.956051 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:49.956465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:49.956897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.056941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 82/161] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 536 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:58:50.057958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.063419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.065206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.167375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.179938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1432e88-f118-464b-b007-f01484f6c48f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.215256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5e6ac4-2869-4836-be61-cc7a22551b28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.216042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5e6ac4-2869-4836-be61-cc7a22551b28 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.216464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5e6ac4-2869-4836-be61-cc7a22551b28 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.216825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5e6ac4-2869-4836-be61-cc7a22551b28 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.227795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5e6ac4-2869-4836-be61-cc7a22551b28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-de5e6ac4-2869-4836-be61-cc7a22551b28 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:50.232865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:50.233835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:50.234684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 83/162] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/roles/admin => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:50.237321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21089c0-e298-4584-8df4-3edeacb72cc6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.237321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21089c0-e298-4584-8df4-3edeacb72cc6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.237321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21089c0-e298-4584-8df4-3edeacb72cc6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.238226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 84/163] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:50.238994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21089c0-e298-4584-8df4-3edeacb72cc6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.276041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 80/164] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:49 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 574 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:50.276999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.279387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.346828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21089c0-e298-4584-8df4-3edeacb72cc6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.381888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd42878-1151-4b67-b1cf-f0aa1b49dc35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.382812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd42878-1151-4b67-b1cf-f0aa1b49dc35 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.383230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd42878-1151-4b67-b1cf-f0aa1b49dc35 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.383680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd42878-1151-4b67-b1cf-f0aa1b49dc35 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.384929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd42878-1151-4b67-b1cf-f0aa1b49dc35 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.386659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dd42878-1151-4b67-b1cf-f0aa1b49dc35 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.388689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 85/165] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.390029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12646e7-8da2-468f-9aea-3723348a6a1c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.390458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12646e7-8da2-468f-9aea-3723348a6a1c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.390867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e12646e7-8da2-468f-9aea-3723348a6a1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.410700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e12646e7-8da2-468f-9aea-3723348a6a1c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:50.412167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e12646e7-8da2-468f-9aea-3723348a6a1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:50.421096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.450298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2519e00-e790-4a78-90f4-c50d283e2930 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.488951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.489568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.489839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.490137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.490795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.491096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.493094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.502957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 81/166] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.504228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.515777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 86/167] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:50.516898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12646e7-8da2-468f-9aea-3723348a6a1c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.619851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.678385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c342469-5038-491f-bb0f-215b607ad3e4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.732379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.733024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.733296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.733564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.735103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.736556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.740181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 82/168] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.741116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.776424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e12646e7-8da2-468f-9aea-3723348a6a1c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.869506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4941389-ac71-4b4a-aa31-96980006cd27 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:50.898269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.898974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.899201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.899474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.900221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] RBAC: Adding query filter params (role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=9564038bafec44429094752520f2166c) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:50.900525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.901981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.918130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 87/169] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/role_assignments?user.id=9564038bafec44429094752520f2166c&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 608 bytes in 401 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.919003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:50.924848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.936002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:50.941803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:50.942490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:50.942768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:50.943042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] PATH_INFO: `/v3/users/4aa345955ed042db9837b36c3d6532b0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:50.944184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] RBAC: Authorizing `identity:get_user(user_id=4aa345955ed042db9837b36c3d6532b0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:50.945668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:50.958600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 83/170] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/users/4aa345955ed042db9837b36c3d6532b0 => generated 247 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:50.959659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.186773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c54648c-2121-46df-a4fc-8c09d490ffa8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.232312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b2903-c2ad-4db6-9a6d-e175d12c7539 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.271667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.272682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.273166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.273644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.274764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=9564038bafec44429094752520f2166c) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:51.275314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.277113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.290159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 88/171] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/role_assignments?user.id=9564038bafec44429094752520f2166c&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 608 bytes in 371 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:51.291511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.324135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64345f52-857c-4f5f-96e0-29d163939f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.324795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64345f52-857c-4f5f-96e0-29d163939f89 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.325075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64345f52-857c-4f5f-96e0-29d163939f89 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.325350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64345f52-857c-4f5f-96e0-29d163939f89 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.462289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.494387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.500119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64345f52-857c-4f5f-96e0-29d163939f89 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.501632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64345f52-857c-4f5f-96e0-29d163939f89 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.521206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe175e93-f098-4d5e-b86b-7b9f1cd94514 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.589184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbc4f84-4c65-48bb-9919-f7ca2428554d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.590276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc4f84-4c65-48bb-9919-f7ca2428554d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.590769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc4f84-4c65-48bb-9919-f7ca2428554d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.591235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbc4f84-4c65-48bb-9919-f7ca2428554d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.627020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbc4f84-4c65-48bb-9919-f7ca2428554d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ccbc4f84-4c65-48bb-9919-f7ca2428554d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:51.632838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:51.633782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:51.634605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:51.642196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 89/172] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 351 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:51.643788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782f9e94-9172-4c30-be62-f9ed1b7fac02 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.644273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782f9e94-9172-4c30-be62-f9ed1b7fac02 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.644765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782f9e94-9172-4c30-be62-f9ed1b7fac02 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.646343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 90/173] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:51.651403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782f9e94-9172-4c30-be62-f9ed1b7fac02 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.744445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 84/174] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 785 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:51.745717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f20bb-0057-4b00-bdb6-075e58bd5308 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.746020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f20bb-0057-4b00-bdb6-075e58bd5308 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.746289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a89f20bb-0057-4b00-bdb6-075e58bd5308 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.776292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a89f20bb-0057-4b00-bdb6-075e58bd5308 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:51.777495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a89f20bb-0057-4b00-bdb6-075e58bd5308 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:51.850525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782f9e94-9172-4c30-be62-f9ed1b7fac02 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:51.877439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 85/175] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:51.878783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f20bb-0057-4b00-bdb6-075e58bd5308 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:51.921535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.935966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:51.936606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:51.936882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:51.937154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:51.937842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:51.938190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:51.939600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:51.943765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:51.955824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 91/176] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 305 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:51.963040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.133943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a89f20bb-0057-4b00-bdb6-075e58bd5308 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.179737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a886d99-9389-491a-8dbb-7aa6ffc5de87 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.195331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.196317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.196744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.197157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.199136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.200763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.202618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 86/177] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:52.217149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.246415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.247384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.247814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.248173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] PATH_INFO: `/v3/users/9564038bafec44429094752520f2166c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.249403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] RBAC: Authorizing `identity:get_user(user_id=9564038bafec44429094752520f2166c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.251138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.253216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 92/178] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:51 2022] GET /identity/v3/users/9564038bafec44429094752520f2166c => generated 275 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:52.254371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.326216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.335484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.340083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fdf2902-6559-473f-929f-019cad5ba6fb None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.377199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5cd57d1-e899-4c3a-8081-3a5fa553c528 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.377758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c6d6089-7390-4ff6-86fb-3e5841ff0840 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.378272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cd57d1-e899-4c3a-8081-3a5fa553c528 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.378693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cd57d1-e899-4c3a-8081-3a5fa553c528 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.379105 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5cd57d1-e899-4c3a-8081-3a5fa553c528 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.380101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5cd57d1-e899-4c3a-8081-3a5fa553c528 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=4aa345955ed042db9837b36c3d6532b0) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:52.380649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5cd57d1-e899-4c3a-8081-3a5fa553c528 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.382368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5cd57d1-e899-4c3a-8081-3a5fa553c528 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.391320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 87/179] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/role_assignments?user.id=4aa345955ed042db9837b36c3d6532b0&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:52.392857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b22e706-71d1-47b4-b75b-a53f4014f0fc None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.393336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b22e706-71d1-47b4-b75b-a53f4014f0fc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.393813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b22e706-71d1-47b4-b75b-a53f4014f0fc None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.395337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 88/180] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:52.396269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b22e706-71d1-47b4-b75b-a53f4014f0fc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.431034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8727b5-9204-4617-90ed-08d65a2f1b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.431694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8727b5-9204-4617-90ed-08d65a2f1b4e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.431982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8727b5-9204-4617-90ed-08d65a2f1b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.435147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8727b5-9204-4617-90ed-08d65a2f1b4e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.460030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef8727b5-9204-4617-90ed-08d65a2f1b4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ef8727b5-9204-4617-90ed-08d65a2f1b4e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:52.471945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:52.472941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:52.473736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 93/181] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:52.475179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4506692b-b40a-4b79-bb59-9c5513d3c656 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.475288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4506692b-b40a-4b79-bb59-9c5513d3c656 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.476443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4506692b-b40a-4b79-bb59-9c5513d3c656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.514708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4506692b-b40a-4b79-bb59-9c5513d3c656 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:52.515943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4506692b-b40a-4b79-bb59-9c5513d3c656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:52.615397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b22e706-71d1-47b4-b75b-a53f4014f0fc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.655792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 94/182] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:52.656962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4506692b-b40a-4b79-bb59-9c5513d3c656 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.659729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.660357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.660690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.661860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.701471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.723564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.756797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.759534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.792967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4506692b-b40a-4b79-bb59-9c5513d3c656 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:52.834933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216b6d5d-b27f-4a6e-9eb1-71bd7f848679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:52.835762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216b6d5d-b27f-4a6e-9eb1-71bd7f848679 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.836158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216b6d5d-b27f-4a6e-9eb1-71bd7f848679 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.836535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216b6d5d-b27f-4a6e-9eb1-71bd7f848679 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.837384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216b6d5d-b27f-4a6e-9eb1-71bd7f848679 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:52.837794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216b6d5d-b27f-4a6e-9eb1-71bd7f848679 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:52.839365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216b6d5d-b27f-4a6e-9eb1-71bd7f848679 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:52.848540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 95/183] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:52.849896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ed8a6a-2dd1-42e7-91c3-b5318817cffa None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.850341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ed8a6a-2dd1-42e7-91c3-b5318817cffa None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.850712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4ed8a6a-2dd1-42e7-91c3-b5318817cffa None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.852013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 96/184] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:52.859337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48c00d-5588-4a50-be9f-bcce93bb4c1c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:52.859615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48c00d-5588-4a50-be9f-bcce93bb4c1c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:52.859886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f48c00d-5588-4a50-be9f-bcce93bb4c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:52.879576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9f48c00d-5588-4a50-be9f-bcce93bb4c1c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:52.880705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9f48c00d-5588-4a50-be9f-bcce93bb4c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:52.894131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 89/185] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:52.895576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:52.965059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:52.971771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 97/186] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:52.972459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f48c00d-5588-4a50-be9f-bcce93bb4c1c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.017772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.020028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ea26af-8853-473c-979b-50abc039ee87 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.060351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f42e2a3-a5d2-432f-b5db-499500bb567c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.060998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42e2a3-a5d2-432f-b5db-499500bb567c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.061276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42e2a3-a5d2-432f-b5db-499500bb567c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.061532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42e2a3-a5d2-432f-b5db-499500bb567c None admin] PATH_INFO: `/v3/users/910a16abec7e4757826385e4656f01d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.075038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f42e2a3-a5d2-432f-b5db-499500bb567c None admin] RBAC: Authorizing `identity:get_user(user_id=910a16abec7e4757826385e4656f01d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.076536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f42e2a3-a5d2-432f-b5db-499500bb567c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.078236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 90/187] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/users/910a16abec7e4757826385e4656f01d7 => generated 283 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.079388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a527a87-9baf-4f74-ac00-e4196d180b65 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.079667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a527a87-9baf-4f74-ac00-e4196d180b65 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.079940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a527a87-9baf-4f74-ac00-e4196d180b65 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.081131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 91/188] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:53 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:53.081682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f48c00d-5588-4a50-be9f-bcce93bb4c1c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.081835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a527a87-9baf-4f74-ac00-e4196d180b65 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.118533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.119489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.119959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.120402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] PATH_INFO: `/v3/projects/d959227356ef497d882b4ac75d6c2557` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.121701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] RBAC: Authorizing `identity:get_project(project_id=d959227356ef497d882b4ac75d6c2557)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.123253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.125077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 98/189] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:52 2022] GET /identity/v3/projects/d959227356ef497d882b4ac75d6c2557 => generated 312 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.125647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.188267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a527a87-9baf-4f74-ac00-e4196d180b65 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.211163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.226787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ec76c6-6c6c-49f6-8db7-8cd6dcb6ce4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.227356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ec76c6-6c6c-49f6-8db7-8cd6dcb6ce4e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.227719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ec76c6-6c6c-49f6-8db7-8cd6dcb6ce4e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.228067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ec76c6-6c6c-49f6-8db7-8cd6dcb6ce4e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.233764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-039578f3-76d3-405e-bf2f-450ea8b88a6b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.238159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ec76c6-6c6c-49f6-8db7-8cd6dcb6ce4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a6ec76c6-6c6c-49f6-8db7-8cd6dcb6ce4e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:53.242967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:53.244100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:58:53.245061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:58:53.248655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.249276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 92/190] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/roles/reader => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:53.250542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b320f58b-cfe5-4c48-8848-aac2c7f1aa1e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.250730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b320f58b-cfe5-4c48-8848-aac2c7f1aa1e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.251002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b320f58b-cfe5-4c48-8848-aac2c7f1aa1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.269089 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b320f58b-cfe5-4c48-8848-aac2c7f1aa1e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:53.270344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b320f58b-cfe5-4c48-8848-aac2c7f1aa1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:53.272985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.273590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.273863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.274156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.336454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 93/191] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:53.337704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b320f58b-cfe5-4c48-8848-aac2c7f1aa1e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.345982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.347465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.440424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b320f58b-cfe5-4c48-8848-aac2c7f1aa1e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.451651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.454954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 99/192] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:53.456017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.467946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.477183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.478284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.478570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.478941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.479555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=d959227356ef497d882b4ac75d6c2557, user.id=9564038bafec44429094752520f2166c) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:53.479855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.481344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.487719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 94/193] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/role_assignments?user.id=9564038bafec44429094752520f2166c&scope.project.id=d959227356ef497d882b4ac75d6c2557&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.488547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.583843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f930dd2d-266c-474f-8265-f6e26daa5918 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.619093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804be08c-fdc4-4dd8-808e-89b14174d21b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.622554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.623208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.623483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.623753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.624433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:53.624733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.626170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.640307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 100/194] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.640307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.666588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.667398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.667887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.667993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.669155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.670856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.672388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 95/195] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/domains/default => generated 199 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.674880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.718811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.726294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.755396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199a3983-715f-43e0-9548-682ff652fd30 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.786309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-347decd0-d227-45fc-b1d7-44248b8249fc None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.787919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.788556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.788831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.789098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.790282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.791729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.793398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 101/196] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.793398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.819205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.819813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.820087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.820358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.830557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:53.835870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:53.836752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 96/197] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:53.838498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.898617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e0dcf9-17fb-4097-b1f5-82f8918826c8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.931989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.932681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.932961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.933244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] PATH_INFO: `/v3/users/925a76c63ad5415097d7368c883d5c62` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.934405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] RBAC: Authorizing `identity:get_user(user_id=925a76c63ad5415097d7368c883d5c62)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.935852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:53.935977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3781957b-b394-437b-a59e-4ae2a97860f7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:53.937433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 102/198] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/users/925a76c63ad5415097d7368c883d5c62 => generated 289 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:53.937557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.938210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:53.949206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:53.975169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:53.975779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:53.976053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:53.976321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:53.976966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:53.978538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.000197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 97/199] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:53 2022] POST /identity/v3/users => generated 297 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:54.000911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.045964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-189f545e-05f7-47d0-9902-a53613868d14 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.086204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.087186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.087467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.087741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.088495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=910a16abec7e4757826385e4656f01d7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.088802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.090329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.102217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 103/200] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:53 2022] GET /identity/v3/role_assignments?user.id=910a16abec7e4757826385e4656f01d7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.103647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.118199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec6899b-3b71-4093-b90e-9ce904d39e71 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.159092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.160160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.160650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.161168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.162194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:54.162727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.164482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.174682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 98/201] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:54.182213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.209338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.214885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.260145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac0de40f-a6df-4d0f-8929-c31063788b44 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.315902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.316787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.317065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.317335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.326380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cb9755-211a-49a5-be7b-b6924a77ee3e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.372988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.373720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.373908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.374215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] PATH_INFO: `/v3/users/910a16abec7e4757826385e4656f01d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.375471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] RBAC: Authorizing `identity:get_user(user_id=910a16abec7e4757826385e4656f01d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.378033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.378943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 99/202] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/users/910a16abec7e4757826385e4656f01d7 => generated 283 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:54.384818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.403382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.405186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.495586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.527987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 104/203] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:54.533656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.551747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.579892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d2c2940-3d22-454e-a140-24a2132ed1bf None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.618135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.618764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.619043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.619312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.713656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.715179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.760616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc14fe2-4d3e-4394-8d4b-f5a5d24a20c3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:54.815279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:54.831852 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 100/204] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 447 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:58:54.835634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:54.837394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.840409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:54.843109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:54.843605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:54.848182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:54.849959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:54.852492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 105/205] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:54.858558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:54.950365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:55.034881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad4ea96-6161-449a-abac-f4bd000414e6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:55.066321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd94422-1bef-44c5-b155-5060e0b87dab None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:55.114122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:55.114122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:55.114122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:55.114122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:55.114511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:55.114595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d56dd9-0ab8-4ed6-a4ed-5162e7a48d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:55.114677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:55.114745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d56dd9-0ab8-4ed6-a4ed-5162e7a48d77 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:55.114745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d56dd9-0ab8-4ed6-a4ed-5162e7a48d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:55.114745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d56dd9-0ab8-4ed6-a4ed-5162e7a48d77 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:55.114920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 101/206] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:55.114984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d56dd9-0ab8-4ed6-a4ed-5162e7a48d77 None admin] RBAC: Adding query filter params (role.id=837dd36862a94b73aa5d4d06027541fc, scope.project.id=507e44cb9594402a9fd0ac69d80cd885, user.id=925a76c63ad5415097d7368c883d5c62) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:55.115048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d56dd9-0ab8-4ed6-a4ed-5162e7a48d77 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:55.116709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d56dd9-0ab8-4ed6-a4ed-5162e7a48d77 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:55.122953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:55.123855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 106/207] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:54 2022] GET /identity/v3/role_assignments?user.id=925a76c63ad5415097d7368c883d5c62&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=837dd36862a94b73aa5d4d06027541fc => generated 608 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:55.188763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:55.237205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e1096f8-4fbd-4b06-b118-aade7aea1968 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:55.292001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83954c0a-4a74-4755-a320-90bc5fcc9d48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:55.292635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83954c0a-4a74-4755-a320-90bc5fcc9d48 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:55.292913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83954c0a-4a74-4755-a320-90bc5fcc9d48 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:55.293182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83954c0a-4a74-4755-a320-90bc5fcc9d48 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:55.294014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83954c0a-4a74-4755-a320-90bc5fcc9d48 None admin] RBAC: Adding query filter params (role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=910a16abec7e4757826385e4656f01d7) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:55.294324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83954c0a-4a74-4755-a320-90bc5fcc9d48 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:55.295806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83954c0a-4a74-4755-a320-90bc5fcc9d48 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:55.306901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 102/208] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:55 2022] GET /identity/v3/role_assignments?user.id=910a16abec7e4757826385e4656f01d7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 259 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:56.538053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff32381-5e5f-4288-9c85-ecf5a983a53b None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.538053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff32381-5e5f-4288-9c85-ecf5a983a53b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.538053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff32381-5e5f-4288-9c85-ecf5a983a53b None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.548016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 107/209] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:56 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:56.553815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d0860-c602-41af-b7dd-e6546874641d None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.553815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d0860-c602-41af-b7dd-e6546874641d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.553815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d0860-c602-41af-b7dd-e6546874641d None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.553815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 103/210] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:56.562559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a929acd9-2d70-408a-bf53-600bd49a1d32 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.562722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4238d6-dc8f-4f91-a458-a21e60dbcb2a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.562842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4238d6-dc8f-4f91-a458-a21e60dbcb2a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.562990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a929acd9-2d70-408a-bf53-600bd49a1d32 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.563108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4238d6-dc8f-4f91-a458-a21e60dbcb2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.563352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a929acd9-2d70-408a-bf53-600bd49a1d32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.582731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a929acd9-2d70-408a-bf53-600bd49a1d32 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:56.584072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a929acd9-2d70-408a-bf53-600bd49a1d32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:56.586925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ec4238d6-dc8f-4f91-a458-a21e60dbcb2a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:56.588311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ec4238d6-dc8f-4f91-a458-a21e60dbcb2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:56.674025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 104/211] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:56.674231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 108/212] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:56.701324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59029942-edd4-4094-b39e-f07c9efd4a5a None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.701324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59029942-edd4-4094-b39e-f07c9efd4a5a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.701324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59029942-edd4-4094-b39e-f07c9efd4a5a None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.702106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 105/213] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:56.715578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22ed1d-9c1e-4a6f-abcc-c7726d9758ad None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:56.715857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22ed1d-9c1e-4a6f-abcc-c7726d9758ad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:56.716127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c22ed1d-9c1e-4a6f-abcc-c7726d9758ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:56.738872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4c22ed1d-9c1e-4a6f-abcc-c7726d9758ad None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:56.740040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4c22ed1d-9c1e-4a6f-abcc-c7726d9758ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:56.770429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:56.814132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 109/214] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:56.932840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59029942-edd4-4094-b39e-f07c9efd4a5a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:56.980301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c22ed1d-9c1e-4a6f-abcc-c7726d9758ad None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:56.985483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.039966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59029942-edd4-4094-b39e-f07c9efd4a5a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.079615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.080651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.081113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.081626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.083066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.084767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.091403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 106/215] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:58:56 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.092514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.155400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c22ed1d-9c1e-4a6f-abcc-c7726d9758ad None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.193658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.194328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.194709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.194977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.198670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b459efee-f1bb-4546-8f94-c5b2a5f9535c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.206140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:57.211160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:57.212182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 110/216] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:58:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:57.217842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.224603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.243423 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.245826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.247399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.247936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.248722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.249660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.251856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.290691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 107/217] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:58:57 2022] POST /identity/v3/users => generated 297 bytes in 198 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:58:57.292050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.342239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4258c1-56f0-4179-9464-59885f8dda9d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.393340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.393992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.394284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.394548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.412807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:57.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:57.422300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:58:57.423111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:58:57.423724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 111/218] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/roles/member => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:58:57.430688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.441382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49518bd9-5545-411c-8c48-48415b275a72 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.491727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.492376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.492671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.492911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.493835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:57.494230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.495682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.505780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 108/219] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.511341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.559028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.575282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.577975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ca9046-8f85-46ce-bf2b-1e7c8e1712a9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.626419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286f9ea0-281a-48f5-8658-fc33ae6c92d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.627680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286f9ea0-281a-48f5-8658-fc33ae6c92d6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.628306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286f9ea0-281a-48f5-8658-fc33ae6c92d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.628848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286f9ea0-281a-48f5-8658-fc33ae6c92d6 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.629686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286f9ea0-281a-48f5-8658-fc33ae6c92d6 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:57.630437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286f9ea0-281a-48f5-8658-fc33ae6c92d6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.632647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286f9ea0-281a-48f5-8658-fc33ae6c92d6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.643878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 112/220] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.646035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca593350-79d2-4a1c-a8bd-e868148e110b None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.646328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca593350-79d2-4a1c-a8bd-e868148e110b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.646593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca593350-79d2-4a1c-a8bd-e868148e110b None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.647966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 113/221] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:58:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:58:57.649922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca593350-79d2-4a1c-a8bd-e868148e110b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.653762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9f02a38-67a8-49e6-8a59-e4b73d3c9dc7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.701568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-731ebcb8-127a-4634-ba1b-82bd9b1b47ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.702362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ebcb8-127a-4634-ba1b-82bd9b1b47ef None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.702641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ebcb8-127a-4634-ba1b-82bd9b1b47ef None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.702912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-731ebcb8-127a-4634-ba1b-82bd9b1b47ef None admin] PATH_INFO: `/v3/users/910a16abec7e4757826385e4656f01d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.704161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ebcb8-127a-4634-ba1b-82bd9b1b47ef None admin] RBAC: Authorizing `identity:get_user(user_id=910a16abec7e4757826385e4656f01d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.705729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-731ebcb8-127a-4634-ba1b-82bd9b1b47ef None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.710133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 109/222] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/users/910a16abec7e4757826385e4656f01d7 => generated 283 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:58:57.710133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e52c88-22dd-4e7a-9118-4ecfe4b26c50 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.710133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e52c88-22dd-4e7a-9118-4ecfe4b26c50 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.710133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e52c88-22dd-4e7a-9118-4ecfe4b26c50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.730209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b9e52c88-22dd-4e7a-9118-4ecfe4b26c50 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:58:57.731390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b9e52c88-22dd-4e7a-9118-4ecfe4b26c50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:58:57.776323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca593350-79d2-4a1c-a8bd-e868148e110b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.810412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.811089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.811360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.811668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] PATH_INFO: `/v3/users/89151d2686a240ec86c5a78785cdabb1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.816843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 110/223] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:58:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:58:57.818039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e52c88-22dd-4e7a-9118-4ecfe4b26c50 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.843679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] RBAC: Authorizing `identity:get_user(user_id=89151d2686a240ec86c5a78785cdabb1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.845166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.853418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 114/224] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/users/89151d2686a240ec86c5a78785cdabb1 => generated 297 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.854915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:57.860571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.868418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:57.935039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e52c88-22dd-4e7a-9118-4ecfe4b26c50 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:57.972548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:57.973664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:57.974077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:57.974506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:57.975786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:57.977630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:57.979766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 111/225] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:57.985426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.089473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c123fd-f667-450f-b020-f6ffd5d640d7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.099333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fff773-52d1-48ac-9ece-49dca964e829 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.102359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.152778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.154170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.155946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.156425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2/users/910a16abec7e4757826385e4656f01d7/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.160663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] RBAC: Authorizing `identity:create_grant(project_id=bb140e38668841f4a920514f6d6826f2, user_id=910a16abec7e4757826385e4656f01d7, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.163446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.171369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.173364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.173364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.173364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.193476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 112/226] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:57 2022] PUT /identity/v3/projects/bb140e38668841f4a920514f6d6826f2/users/910a16abec7e4757826385e4656f01d7/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:58:58.204668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.327179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.328682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.332685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a4c2414-d79f-4467-9b4b-50b2a8243e8c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.335980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.379278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.380421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.380941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.381411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.396011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:58:58.402058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:58:58.403002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:58:58.403813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:58:58.406930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 113/227] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:58:58.419466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.459967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.519731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 115/228] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:58:57 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 665 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:58:58.525379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.580707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed45f2fe-bddb-4abd-9e3d-3ae33c9afb66 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.615863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.616482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.616735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.616999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.617663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:58.617956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.619480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.633098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 114/229] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:58.634489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.657694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-529003ab-c585-4e61-95e4-defedf088b23 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.661110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.700855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.702123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.702687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.703139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.704434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.706101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.708104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 116/230] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:58.714988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.726284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:58.743007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c00a2e-4daf-4e18-bc03-98b35d0de59c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.787431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.788093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.788365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.788629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] PATH_INFO: `/v3/users/910a16abec7e4757826385e4656f01d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.793682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] RBAC: Authorizing `identity:get_user(user_id=910a16abec7e4757826385e4656f01d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.793682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.793682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 115/231] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/users/910a16abec7e4757826385e4656f01d7 => generated 283 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:58.799458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:58.910031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e276ba-294a-4ccc-9f62-697936449595 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:58.953963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:58.955449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:58.955726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:58.955997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:58.957155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=89151d2686a240ec86c5a78785cdabb1) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:58:58.957463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:58.959013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:58.973129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 117/232] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/role_assignments?user.id=89151d2686a240ec86c5a78785cdabb1&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 259 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:59.014769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f3e178-84b0-4c0d-a06a-c4124ffb583e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.080255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-091252cb-494a-49d7-9a05-e0c187ec988a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.080918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091252cb-494a-49d7-9a05-e0c187ec988a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.081190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091252cb-494a-49d7-9a05-e0c187ec988a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.081456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-091252cb-494a-49d7-9a05-e0c187ec988a None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.082712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091252cb-494a-49d7-9a05-e0c187ec988a None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.084201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-091252cb-494a-49d7-9a05-e0c187ec988a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.085933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 116/233] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:58:58 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:58:59.092454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:58:59.157073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:58:59.247385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a7c389b-6a65-4005-898f-1bf09ef53fb9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:58:59.294364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff431de2-6a6e-4478-93d7-0a97fccd3930 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:58:59.295052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff431de2-6a6e-4478-93d7-0a97fccd3930 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:58:59.295318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff431de2-6a6e-4478-93d7-0a97fccd3930 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:58:59.295596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff431de2-6a6e-4478-93d7-0a97fccd3930 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2/users/910a16abec7e4757826385e4656f01d7/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:58:59.304004 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff431de2-6a6e-4478-93d7-0a97fccd3930 None admin] RBAC: Authorizing `identity:create_grant(project_id=bb140e38668841f4a920514f6d6826f2, user_id=910a16abec7e4757826385e4656f01d7, role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:58:59.306869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff431de2-6a6e-4478-93d7-0a97fccd3930 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:58:59.336798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 118/234] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:58:59 2022] PUT /identity/v3/projects/bb140e38668841f4a920514f6d6826f2/users/910a16abec7e4757826385e4656f01d7/roles/1b55a134a7b24115b2bb8691600d2619 => generated 0 bytes in 245 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:00.179672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8657caf8-338a-4277-a453-70381c6245ae None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.180577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8657caf8-338a-4277-a453-70381c6245ae None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.180577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8657caf8-338a-4277-a453-70381c6245ae None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.182177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 117/235] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:00.187470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9720f1-d474-4a5e-a20d-df0cf7b67c35 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.187904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9720f1-d474-4a5e-a20d-df0cf7b67c35 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.188097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9720f1-d474-4a5e-a20d-df0cf7b67c35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.209995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cc9720f1-d474-4a5e-a20d-df0cf7b67c35 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:00.211204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cc9720f1-d474-4a5e-a20d-df0cf7b67c35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:00.312601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 119/236] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:00.349016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c1730-c328-49ca-aa11-871cf01b9edc None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.349321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c1730-c328-49ca-aa11-871cf01b9edc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.349540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563c1730-c328-49ca-aa11-871cf01b9edc None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.351097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 118/237] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:00.363422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0278058c-e01b-454e-875c-78842b748502 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.363701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0278058c-e01b-454e-875c-78842b748502 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.363970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0278058c-e01b-454e-875c-78842b748502 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.382424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0278058c-e01b-454e-875c-78842b748502 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:00.383588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0278058c-e01b-454e-875c-78842b748502 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:00.404395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:00.476578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 120/238] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:00.697031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c1730-c328-49ca-aa11-871cf01b9edc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.727065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:00.848525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563c1730-c328-49ca-aa11-871cf01b9edc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:00.913186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0278058c-e01b-454e-875c-78842b748502 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:00.926530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:00.927533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:00.927943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:00.928345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:00.957714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:00.968510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:00.969387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:00.970179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:00.970783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 119/239] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:00.977682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.043416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0278058c-e01b-454e-875c-78842b748502 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.085408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.086474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.086965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.087574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.088878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.090543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.092311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 121/240] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/domains/default => generated 199 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:01.098460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.127916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.164776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.193662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa6bbb7-9478-4a2a-b552-d2a3783b3ad8 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.217356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329e9b7e-3598-4861-a306-0333449a07fe None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.244802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-191ed8d0-84c7-4a7b-8107-fb08c56584af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.245346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ed8d0-84c7-4a7b-8107-fb08c56584af None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.245622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ed8d0-84c7-4a7b-8107-fb08c56584af None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.245889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-191ed8d0-84c7-4a7b-8107-fb08c56584af None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.246835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191ed8d0-84c7-4a7b-8107-fb08c56584af None admin] RBAC: Adding query filter params (name=reader) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:01.247138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191ed8d0-84c7-4a7b-8107-fb08c56584af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.249251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-191ed8d0-84c7-4a7b-8107-fb08c56584af None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.258721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0a3fc1-d258-4750-8d90-ac7845c6277f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.259699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0a3fc1-d258-4750-8d90-ac7845c6277f None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.260161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0a3fc1-d258-4750-8d90-ac7845c6277f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.260614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0a3fc1-d258-4750-8d90-ac7845c6277f None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.261469 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0a3fc1-d258-4750-8d90-ac7845c6277f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.261964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 120/241] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:01.263323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1058ffa6-3b46-477a-a888-6eed9fba8b07 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.263603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1058ffa6-3b46-477a-a888-6eed9fba8b07 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.263872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1058ffa6-3b46-477a-a888-6eed9fba8b07 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.264077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba0a3fc1-d258-4750-8d90-ac7845c6277f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.265057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 121/242] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:01.270021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1058ffa6-3b46-477a-a888-6eed9fba8b07 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.304996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 122/243] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:01 2022] POST /identity/v3/users => generated 293 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:01.304996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ab079-e939-414e-b2cf-e4566a3fc39d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.304996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ab079-e939-414e-b2cf-e4566a3fc39d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.304996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973ab079-e939-414e-b2cf-e4566a3fc39d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.349533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-973ab079-e939-414e-b2cf-e4566a3fc39d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:01.354211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-973ab079-e939-414e-b2cf-e4566a3fc39d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:01.475403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1058ffa6-3b46-477a-a888-6eed9fba8b07 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.506764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 123/244] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 204 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:01.531106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e873fa2-09d3-469a-b848-6e5e53dfd8fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.531106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e873fa2-09d3-469a-b848-6e5e53dfd8fe None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.531106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e873fa2-09d3-469a-b848-6e5e53dfd8fe None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.531106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e873fa2-09d3-469a-b848-6e5e53dfd8fe None admin] PATH_INFO: `/v3/users/fa76bf5317684688a052c454c37fdfd9` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.546401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.547915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e873fa2-09d3-469a-b848-6e5e53dfd8fe None admin] RBAC: Authorizing `identity:get_user(user_id=fa76bf5317684688a052c454c37fdfd9)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.549465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e873fa2-09d3-469a-b848-6e5e53dfd8fe None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.551284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 122/245] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/users/fa76bf5317684688a052c454c37fdfd9 => generated 297 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:01.558263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ab079-e939-414e-b2cf-e4566a3fc39d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.582483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.582853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d1c68d-662e-4d96-a5fc-ab5e1e6b8000 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.583138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d1c68d-662e-4d96-a5fc-ab5e1e6b8000 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.583406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70d1c68d-662e-4d96-a5fc-ab5e1e6b8000 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.584784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 123/246] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:01.590793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96132002-ec10-48be-8f39-4bab3df287a7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.591081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96132002-ec10-48be-8f39-4bab3df287a7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.591352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96132002-ec10-48be-8f39-4bab3df287a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.612362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-96132002-ec10-48be-8f39-4bab3df287a7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:01.613560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-96132002-ec10-48be-8f39-4bab3df287a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:01.676134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973ab079-e939-414e-b2cf-e4566a3fc39d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.694118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 124/247] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:01.716483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.717105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.717429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.717703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:01.790667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:01.792043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:01.793886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96132002-ec10-48be-8f39-4bab3df287a7 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.807135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.919008 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 124/248] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:59:01.924901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:01.944788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96132002-ec10-48be-8f39-4bab3df287a7 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:01.981940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:01.987789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:01.989531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:01.989531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:01.989531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.002393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:02.010763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:02.011686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:02.012459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 125/249] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:02.017765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.083788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775708be-2c4a-4c50-b0aa-3f9d0019072d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.096280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.124204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.124849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.125138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.125389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.126536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.128260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.130129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 125/250] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:01 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.130933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.139717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-364aa850-3df7-47b7-91d3-68851504b2aa None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.190252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.191679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.192425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.192877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.193887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.195131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.197991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.210062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 126/251] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:02.211344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.248019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e173ece5-1948-4c93-a308-8c87f6b9a034 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.286925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.342850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4131a3ba-f970-437d-ac51-2bcd84be9ee1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.379991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.381043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.381497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.381942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.382953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] RBAC: Adding query filter params (role.id=837dd36862a94b73aa5d4d06027541fc, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=fa76bf5317684688a052c454c37fdfd9) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.383434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.385171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.386748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf31609e-e67f-4e80-89ed-0f692fecca6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.387406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf31609e-e67f-4e80-89ed-0f692fecca6b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.387825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf31609e-e67f-4e80-89ed-0f692fecca6b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.387946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf31609e-e67f-4e80-89ed-0f692fecca6b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.392272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 127/252] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/role_assignments?user.id=fa76bf5317684688a052c454c37fdfd9&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=837dd36862a94b73aa5d4d06027541fc => generated 259 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.393629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.409387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf31609e-e67f-4e80-89ed-0f692fecca6b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-cf31609e-e67f-4e80-89ed-0f692fecca6b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:02.415283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:02.416309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:02.417096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:02.419864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 126/253] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/roles/member => generated 84 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:02.423246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955990ad-3c74-4e3b-9b3f-b744de272e74 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.423688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955990ad-3c74-4e3b-9b3f-b744de272e74 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.424099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-955990ad-3c74-4e3b-9b3f-b744de272e74 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.425683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 127/254] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:02 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:02.426772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955990ad-3c74-4e3b-9b3f-b744de272e74 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.476643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.557533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1084571d-3248-405e-9f52-c9f71296de79 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.586591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.595530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf256ef4-7246-4e3b-ac3c-81357d60a09a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.596200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf256ef4-7246-4e3b-ac3c-81357d60a09a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.596592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf256ef4-7246-4e3b-ac3c-81357d60a09a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.596928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf256ef4-7246-4e3b-ac3c-81357d60a09a None admin] PATH_INFO: `/v3/users/910a16abec7e4757826385e4656f01d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.598213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf256ef4-7246-4e3b-ac3c-81357d60a09a None admin] RBAC: Authorizing `identity:get_user(user_id=910a16abec7e4757826385e4656f01d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.600045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf256ef4-7246-4e3b-ac3c-81357d60a09a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.602042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 128/255] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/users/910a16abec7e4757826385e4656f01d7 => generated 283 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.608312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2e9a6-9c64-44e8-bedd-332aaa745f3e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.608727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2e9a6-9c64-44e8-bedd-332aaa745f3e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.609068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb2e9a6-9c64-44e8-bedd-332aaa745f3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.630583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ceb2e9a6-9c64-44e8-bedd-332aaa745f3e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:02.631885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ceb2e9a6-9c64-44e8-bedd-332aaa745f3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:02.638837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-955990ad-3c74-4e3b-9b3f-b744de272e74 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.691273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.691885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.692209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.692474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.693654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:02.693965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.696347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.708319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 128/256] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.709165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.719787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 129/257] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:02.720435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb2e9a6-9c64-44e8-bedd-332aaa745f3e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:02.856114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb2e9a6-9c64-44e8-bedd-332aaa745f3e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.898396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.937913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:02.938725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:02.939372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:02.939646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:02.940944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] PATH_INFO: `/v3/users/89151d2686a240ec86c5a78785cdabb1` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:02.941585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] RBAC: Authorizing `identity:get_user(user_id=89151d2686a240ec86c5a78785cdabb1)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:02.943079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:02.946787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf15899-2d20-4d77-baa4-1ffdc628f370 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:02.948735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 130/258] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/users/89151d2686a240ec86c5a78785cdabb1 => generated 297 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:02.950795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.005512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.006840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.007119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.007386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.101225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb3a17f4-4b20-42d6-9e01-087f0e84a4df None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.145949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.147361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.165007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.165731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.166020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.166288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.169577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.171069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.172899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 131/259] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:03.174581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.262651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:03.313806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 129/260] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 605 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:03.314671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.315437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:03.349661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-130f2cf5-be6e-4eda-9e20-c66285963034 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.391069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.391706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.391978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.392241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.414701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:03.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:03.420478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:03.421715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 132/261] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:03.422410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.466162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dcc1e7-6c32-4992-9de5-4b7f12f3fe4a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.510904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.511621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.511897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.512168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2/users/89151d2686a240ec86c5a78785cdabb1/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.514514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] RBAC: Authorizing `identity:create_grant(project_id=bb140e38668841f4a920514f6d6826f2, user_id=89151d2686a240ec86c5a78785cdabb1, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.515980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.548269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 130/262] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:03 2022] PUT /identity/v3/projects/bb140e38668841f4a920514f6d6826f2/users/89151d2686a240ec86c5a78785cdabb1/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:03.549130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.637868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28fc6cd8-eb53-4489-8a90-da71fdd99e0d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.669222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:03.682516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.683517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.683841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.684120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.685562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.687502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.689529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 133/263] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:03.696525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:03.723336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:03.816506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf0572a-9b51-4c79-8917-f45d2926ac1b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.854927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.855544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.855812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.856075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.856801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=910a16abec7e4757826385e4656f01d7) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:03.857103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.858752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.860162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5faf0dab-5229-42d5-b426-6eecb50a8eb2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:03.864971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 134/264] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/role_assignments?user.id=910a16abec7e4757826385e4656f01d7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:03.934077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:03.934682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:03.934963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:03.935233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:03.935917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:03.936217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:03.937766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:03.950412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 131/265] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 401 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:03.956395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.008925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.069410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d64f5fbd-9819-4340-892b-cc8d38a74592 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.105921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.106592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.106863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.107126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] PATH_INFO: `/v3/users/910a16abec7e4757826385e4656f01d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.108252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] RBAC: Authorizing `identity:get_user(user_id=910a16abec7e4757826385e4656f01d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.109712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.111611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 135/266] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:03 2022] GET /identity/v3/users/910a16abec7e4757826385e4656f01d7 => generated 283 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:04.117129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.255466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f58175c0-b4b8-4ba9-80e2-0e036b246b67 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.287268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.299950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.300574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.300848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.301116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.379658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.381290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.499753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 132/267] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:04.505708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.632850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb655549-7549-42ec-8c82-00809edde381 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.644088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.668834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-922b31bd-3817-4a40-aa2b-f0cc5c3d1e72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.669494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922b31bd-3817-4a40-aa2b-f0cc5c3d1e72 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.669785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922b31bd-3817-4a40-aa2b-f0cc5c3d1e72 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.670060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-922b31bd-3817-4a40-aa2b-f0cc5c3d1e72 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.671309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922b31bd-3817-4a40-aa2b-f0cc5c3d1e72 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.672810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-922b31bd-3817-4a40-aa2b-f0cc5c3d1e72 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.674918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 136/268] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:04.681244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:04.729387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:04.802590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bfb019-8198-447d-9170-6179dd7b539c None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.802762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bfb019-8198-447d-9170-6179dd7b539c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.803035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30bfb019-8198-447d-9170-6179dd7b539c None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.804578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 137/269] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:04.810514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8d233-fc68-4845-800f-0fe4903a3b7c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.810796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8d233-fc68-4845-800f-0fe4903a3b7c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.811049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb8d233-fc68-4845-800f-0fe4903a3b7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.841885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3fb8d233-fc68-4845-800f-0fe4903a3b7c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:04.844125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3fb8d233-fc68-4845-800f-0fe4903a3b7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:04.866051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f67dbe-c8b6-4bb9-90b6-c3311d38ad40 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:04.908687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-035053c6-38e0-4daa-b8e4-cd9f87b148b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:04.909312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035053c6-38e0-4daa-b8e4-cd9f87b148b8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.909651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035053c6-38e0-4daa-b8e4-cd9f87b148b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.910135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035053c6-38e0-4daa-b8e4-cd9f87b148b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.910875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035053c6-38e0-4daa-b8e4-cd9f87b148b8 None admin] RBAC: Adding query filter params (role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=910a16abec7e4757826385e4656f01d7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:04.911572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035053c6-38e0-4daa-b8e4-cd9f87b148b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:04.913387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-035053c6-38e0-4daa-b8e4-cd9f87b148b8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:04.923268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 133/270] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:04 2022] GET /identity/v3/role_assignments?user.id=910a16abec7e4757826385e4656f01d7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 608 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:04.924675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257618e3-c6f4-45fb-b048-944aa8e70631 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.925111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257618e3-c6f4-45fb-b048-944aa8e70631 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.925438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257618e3-c6f4-45fb-b048-944aa8e70631 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.927628 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 134/271] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:04.941427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fcc0e8-5b2f-4957-8376-9defb30944f0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:04.941758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fcc0e8-5b2f-4957-8376-9defb30944f0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:04.944168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fcc0e8-5b2f-4957-8376-9defb30944f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:04.944311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 138/272] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:04.962524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f0fcc0e8-5b2f-4957-8376-9defb30944f0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:04.963905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f0fcc0e8-5b2f-4957-8376-9defb30944f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:05.041625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 135/273] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:05.316767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8d233-fc68-4845-800f-0fe4903a3b7c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.332144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fcc0e8-5b2f-4957-8376-9defb30944f0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.360446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.452137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb8d233-fc68-4845-800f-0fe4903a3b7c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.462468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.473230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fcc0e8-5b2f-4957-8376-9defb30944f0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.490606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.491394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.491759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.492029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.493139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.495266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.497174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 139/274] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/domains/default => generated 199 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:05.504688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.513821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b14c65-190f-42a7-9171-be52bdfd5ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.514498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b14c65-190f-42a7-9171-be52bdfd5ad2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.514772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b14c65-190f-42a7-9171-be52bdfd5ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.515041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b14c65-190f-42a7-9171-be52bdfd5ad2 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.525401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b14c65-190f-42a7-9171-be52bdfd5ad2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f0b14c65-190f-42a7-9171-be52bdfd5ad2 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:05.530279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:05.531170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:05.532039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 136/275] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:05.534580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99289410-b481-48ae-88ae-eb7d1a5cd550 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.535222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99289410-b481-48ae-88ae-eb7d1a5cd550 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.535494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99289410-b481-48ae-88ae-eb7d1a5cd550 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.542216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 137/276] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:05 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:05.542216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99289410-b481-48ae-88ae-eb7d1a5cd550 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.624266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cde697b-db47-4465-b41e-efec7efd4566 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.625978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.652262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99289410-b481-48ae-88ae-eb7d1a5cd550 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.659708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94a47e3-b16f-4865-94f1-201d96fc73d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.660312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94a47e3-b16f-4865-94f1-201d96fc73d5 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.660596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94a47e3-b16f-4865-94f1-201d96fc73d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.660867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94a47e3-b16f-4865-94f1-201d96fc73d5 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.661517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94a47e3-b16f-4865-94f1-201d96fc73d5 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.663010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94a47e3-b16f-4865-94f1-201d96fc73d5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.686841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 140/277] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:05 2022] POST /identity/v3/users => generated 293 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:05.687983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8357b1b8-8157-450d-9ba4-91e9c0868b22 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.688268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8357b1b8-8157-450d-9ba4-91e9c0868b22 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.688552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8357b1b8-8157-450d-9ba4-91e9c0868b22 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.704353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.705626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.706167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.706583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.707499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:05.707949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.709533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8357b1b8-8157-450d-9ba4-91e9c0868b22 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:05.709916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.710995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8357b1b8-8157-450d-9ba4-91e9c0868b22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:05.723846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 138/278] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:05.729982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:05.743487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.798048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 141/279] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:05.867748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5af25884-62d5-4d3e-bfe1-60815a4eb44d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:05.906575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:05.907558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:05.908083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:05.908540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] PATH_INFO: `/v3/users/1e5d36533af0452382fc6776816b5b54` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:05.917834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:05.924739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] RBAC: Authorizing `identity:get_user(user_id=1e5d36533af0452382fc6776816b5b54)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:05.926709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:05.933615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 139/280] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/users/1e5d36533af0452382fc6776816b5b54 => generated 293 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:05.935241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8357b1b8-8157-450d-9ba4-91e9c0868b22 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.045413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8357b1b8-8157-450d-9ba4-91e9c0868b22 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.085480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c407dfe-f506-4a60-9a23-f68325bfdd35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.086386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c407dfe-f506-4a60-9a23-f68325bfdd35 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.086795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c407dfe-f506-4a60-9a23-f68325bfdd35 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.087195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c407dfe-f506-4a60-9a23-f68325bfdd35 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.088200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c407dfe-f506-4a60-9a23-f68325bfdd35 None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, user.id=1e5d36533af0452382fc6776816b5b54, scope.system=all) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.088735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c407dfe-f506-4a60-9a23-f68325bfdd35 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.090440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c407dfe-f506-4a60-9a23-f68325bfdd35 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.096347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 142/281] 10.222.0.61 () {56 vars in 1308 bytes} [Thu Sep 29 01:59:05 2022] GET /identity/v3/role_assignments?user.id=1e5d36533af0452382fc6776816b5b54&scope.system=all&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 226 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.108034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.167480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217b040-e254-40a0-911b-1a5d77754d43 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.167664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217b040-e254-40a0-911b-1a5d77754d43 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.167927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7217b040-e254-40a0-911b-1a5d77754d43 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.169532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 143/282] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:06.175246 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c2b019-26e7-40e6-a711-e750070e4b47 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.175556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c2b019-26e7-40e6-a711-e750070e4b47 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.175795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c2b019-26e7-40e6-a711-e750070e4b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.196397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-09c2b019-26e7-40e6-a711-e750070e4b47 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:06.197701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-09c2b019-26e7-40e6-a711-e750070e4b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:06.226428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.236677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c5ef34-430f-47e7-a179-3ac8c37bfa39 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.278553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.279205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.279480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.279752 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.291565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:06.296518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:06.297443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:06.298254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 140/283] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/roles/reader => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:06.303799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.324834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 144/284] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:06.374112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.419567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ccf5c9-a306-4104-9863-522999a69647 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.455143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.455760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.456039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.456313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.456983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:06.457287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.458711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.468069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 141/285] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:06.473884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c2b019-26e7-40e6-a711-e750070e4b47 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.593288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c2b019-26e7-40e6-a711-e750070e4b47 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.627208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.627837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.628112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.628384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] PATH_INFO: `/v3/users/fa76bf5317684688a052c454c37fdfd9` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.629513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] RBAC: Authorizing `identity:get_user(user_id=fa76bf5317684688a052c454c37fdfd9)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.630996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.632753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 145/286] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/users/fa76bf5317684688a052c454c37fdfd9 => generated 297 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.638419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.683213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.734393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.745104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8487a55f-35f0-44f5-9d39-b12785148b1f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.766425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:06.786443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.787085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.787363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.787636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.788891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.790391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.793255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 142/287] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:06.799660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:06.885194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7fd5f1-bc0e-4ac4-88ac-c23b957ac275 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.935489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4522d27-211a-4bcb-8d37-b9928b88fc4d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:06.975145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.975790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.976063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.976330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.977365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:06.978036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:06.978342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:06.978615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2/users/fa76bf5317684688a052c454c37fdfd9/roles/837dd36862a94b73aa5d4d06027541fc` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:06.981563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] RBAC: Authorizing `identity:create_grant(project_id=bb140e38668841f4a920514f6d6826f2, user_id=fa76bf5317684688a052c454c37fdfd9, role_id=837dd36862a94b73aa5d4d06027541fc)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:06.983043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:06.996430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:07.003129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:07.003985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:07.004839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 146/288] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:07.009024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 143/289] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:06 2022] PUT /identity/v3/projects/bb140e38668841f4a920514f6d6826f2/users/fa76bf5317684688a052c454c37fdfd9/roles/837dd36862a94b73aa5d4d06027541fc => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:07.012044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.128166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.148892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0035a4b-6b5e-4b35-b471-5ff2a6f3b74b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.187214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.187755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.188029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.188300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.189042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:07.189346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.190761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.200339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 147/290] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:07.206113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.280536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.352344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22aed09a-f9d9-43bc-ad6d-4854f1826098 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.389366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.390024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.390361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.390641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] PATH_INFO: `/v3/users/89151d2686a240ec86c5a78785cdabb1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.391781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] RBAC: Authorizing `identity:get_user(user_id=89151d2686a240ec86c5a78785cdabb1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.393238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.395188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 144/291] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/users/89151d2686a240ec86c5a78785cdabb1 => generated 297 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:07.400901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.526040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-933fb03a-7f34-4638-b9e2-d31e05f59f17 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.581038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.581658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.581933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.582233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.597855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.655414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.656897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.755796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 148/292] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:59:07.761643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:07.898449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd8c6ad-a7d7-4136-97d4-f6fdb3af77e0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:07.912616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:07.938974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b969eb8d-fce7-49d8-aead-68ec4ca992c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:07.939616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b969eb8d-fce7-49d8-aead-68ec4ca992c8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:07.939942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b969eb8d-fce7-49d8-aead-68ec4ca992c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:07.940279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b969eb8d-fce7-49d8-aead-68ec4ca992c8 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:07.942700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b969eb8d-fce7-49d8-aead-68ec4ca992c8 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:07.944168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b969eb8d-fce7-49d8-aead-68ec4ca992c8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:07.946216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 145/293] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:07.952624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:08.023505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:08.082034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd02eb7-fdaa-483e-b21d-d38dee6fe3ff None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:08.121792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9afe11ae-688a-413e-86af-03485ae0eeed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:08.122783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afe11ae-688a-413e-86af-03485ae0eeed None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.123258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afe11ae-688a-413e-86af-03485ae0eeed None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.123743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9afe11ae-688a-413e-86af-03485ae0eeed None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.124690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afe11ae-688a-413e-86af-03485ae0eeed None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=89151d2686a240ec86c5a78785cdabb1) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:08.125209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afe11ae-688a-413e-86af-03485ae0eeed None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:08.127045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9afe11ae-688a-413e-86af-03485ae0eeed None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:08.133464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 149/294] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:07 2022] GET /identity/v3/role_assignments?user.id=89151d2686a240ec86c5a78785cdabb1&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:08.412650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a0b19f-f725-4b05-a61a-c5d268b45d40 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.412927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a0b19f-f725-4b05-a61a-c5d268b45d40 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.413466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12a0b19f-f725-4b05-a61a-c5d268b45d40 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.415083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 146/295] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:08.424618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54fab1-7286-41eb-aff0-e7a93181233b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.425231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54fab1-7286-41eb-aff0-e7a93181233b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.425721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c54fab1-7286-41eb-aff0-e7a93181233b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.452425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3c54fab1-7286-41eb-aff0-e7a93181233b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:08.453978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3c54fab1-7286-41eb-aff0-e7a93181233b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:08.527258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 150/296] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:08.558688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d934cf-160f-46f9-9d01-89b42667850d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.559314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d934cf-160f-46f9-9d01-89b42667850d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.559647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d934cf-160f-46f9-9d01-89b42667850d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.561054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 147/297] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:08.566524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0b5691-6a90-4001-9486-4d566e670908 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.566810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0b5691-6a90-4001-9486-4d566e670908 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.567141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0b5691-6a90-4001-9486-4d566e670908 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.586789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-aa0b5691-6a90-4001-9486-4d566e670908 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:08.588059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-aa0b5691-6a90-4001-9486-4d566e670908 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:08.623842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:08.661255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 151/298] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:08.842302 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d934cf-160f-46f9-9d01-89b42667850d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:08.911141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0e828c-0fe7-4e31-af88-27cb5f7fcfd5 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.911468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0e828c-0fe7-4e31-af88-27cb5f7fcfd5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.911694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c0e828c-0fe7-4e31-af88-27cb5f7fcfd5 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.913163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 152/299] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:08.918593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf32c714-e4f6-43ca-8e2e-1a1976c1fa06 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.919049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf32c714-e4f6-43ca-8e2e-1a1976c1fa06 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.919328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf32c714-e4f6-43ca-8e2e-1a1976c1fa06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:08.939453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-bf32c714-e4f6-43ca-8e2e-1a1976c1fa06 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:08.940958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bf32c714-e4f6-43ca-8e2e-1a1976c1fa06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:08.950972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d934cf-160f-46f9-9d01-89b42667850d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:08.988149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:08.988786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:08.989055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:08.989323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.000384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:09.005598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:09.006749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 148/300] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:08 2022] GET /identity/v3/roles/reader => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:09.011363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.018842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 153/301] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:09.020028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf32c714-e4f6-43ca-8e2e-1a1976c1fa06 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.055615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.131983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3769b62-0e6d-48b5-a93d-ee676c2f6e2a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.148799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.152577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf32c714-e4f6-43ca-8e2e-1a1976c1fa06 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.176463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.177753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.178447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.178888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.180597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.183130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.185444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 149/302] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:09.193242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.193369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.193369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.193599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.193875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.194587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:09.194887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.196269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.210071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 154/303] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:09.210947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.314148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7931a1eb-b495-4647-be82-382b8170e3ac None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.328104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.328837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d37d14-7420-41db-8dcc-953ac0e46731 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.355094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.356041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.356328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.356879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] PATH_INFO: `/v3/groups` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.357521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.359035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.363071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.363697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.363964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.364227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] PATH_INFO: `/v3/users/9d8cbbd3fe224f6593cc21552efd312d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.386127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 150/304] 10.222.0.61 () {60 vars in 1134 bytes} [Thu Sep 29 01:59:09 2022] POST /identity/v3/groups => generated 229 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:09.386127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.388876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] RBAC: Authorizing `identity:get_user(user_id=9d8cbbd3fe224f6593cc21552efd312d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.390530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.392398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 155/305] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/users/9d8cbbd3fe224f6593cc21552efd312d => generated 293 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:09.400436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.437250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.521503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58af9310-85e0-4d00-a691-3fbed4d1c87c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.552581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59e4c961-e53b-4a74-aa97-6f13f237243f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.566997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.567642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.568215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.568215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.581783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:09.587108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:09.588027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:09.588833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 151/306] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:09.595361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.601582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.602238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.602517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.603237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.603611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] RBAC: Adding query filter params (role.id=837dd36862a94b73aa5d4d06027541fc, user.id=9d8cbbd3fe224f6593cc21552efd312d, scope.system=all) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:09.603925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.605972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 156/307] 10.222.0.61 () {56 vars in 1308 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/role_assignments?user.id=9d8cbbd3fe224f6593cc21552efd312d&scope.system=all&role.id=837dd36862a94b73aa5d4d06027541fc => generated 226 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:09.633085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.717545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e731e5cc-2063-44db-851b-91b74d2fdf60 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.754920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6da38454-71ef-4936-9bff-4ff9f7e9b715 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.755542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da38454-71ef-4936-9bff-4ff9f7e9b715 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.755814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da38454-71ef-4936-9bff-4ff9f7e9b715 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.756077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da38454-71ef-4936-9bff-4ff9f7e9b715 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.756746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da38454-71ef-4936-9bff-4ff9f7e9b715 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:09.757037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da38454-71ef-4936-9bff-4ff9f7e9b715 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.758473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6da38454-71ef-4936-9bff-4ff9f7e9b715 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.768576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 152/308] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:09.774469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:09.878035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:09.895009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50710bc7-027b-4fde-bee2-1d7480b220e6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:09.919616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a7e86f-8222-4270-88eb-9c1aecbfc48c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.920144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a7e86f-8222-4270-88eb-9c1aecbfc48c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.920144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a7e86f-8222-4270-88eb-9c1aecbfc48c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.922912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 153/309] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:09.927297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08158dce-2b12-42d5-9331-773d570670e8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.927578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08158dce-2b12-42d5-9331-773d570670e8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.927854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08158dce-2b12-42d5-9331-773d570670e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.945346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:09.945346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:09.945346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:09.945346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] PATH_INFO: `/v3/users/1e5d36533af0452382fc6776816b5b54` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:09.945709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] RBAC: Authorizing `identity:get_user(user_id=1e5d36533af0452382fc6776816b5b54)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:09.945709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:09.945709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 157/310] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:09 2022] GET /identity/v3/users/1e5d36533af0452382fc6776816b5b54 => generated 293 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:09.947773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-08158dce-2b12-42d5-9331-773d570670e8 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:09.949152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-08158dce-2b12-42d5-9331-773d570670e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:09.951423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.028345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 154/311] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:10.071031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3340d275-fa1f-4b10-bac9-b99017c4b572 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.105473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.106158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.106613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.106772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] PATH_INFO: `/v3/system/users/1e5d36533af0452382fc6776816b5b54/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.108158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=1e5d36533af0452382fc6776816b5b54, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.109545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.121465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 158/312] 10.222.0.61 () {58 vars in 1334 bytes} [Thu Sep 29 01:59:09 2022] PUT /identity/v3/system/users/1e5d36533af0452382fc6776816b5b54/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:59:10.337088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08158dce-2b12-42d5-9331-773d570670e8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.362567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.443562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08158dce-2b12-42d5-9331-773d570670e8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.477662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.478338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.478710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.478977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.489934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:10.494842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:10.495716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:10.496498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 155/313] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/roles/reader => generated 84 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:10.504130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.526042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.616255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66ed797-6af6-45fa-ab8e-574634c980ef None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.651757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.652460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.652729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.653013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.653673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:10.653966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.655394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.664456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 159/314] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:10.670044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.759258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.777235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d861834-ed05-41f5-9681-9cc6e5130775 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.810898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.811533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.811805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.812076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] PATH_INFO: `/v3/users/fa76bf5317684688a052c454c37fdfd9` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:10.813187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] RBAC: Authorizing `identity:get_user(user_id=fa76bf5317684688a052c454c37fdfd9)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:10.814680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:10.821934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:10.822848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 156/315] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/users/fa76bf5317684688a052c454c37fdfd9 => generated 297 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:10.901853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:10.924229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1851df65-be81-4e04-86cf-33863a53a808 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:10.963749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:10.964362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:10.964634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:10.964902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.032449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.033921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.114037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.137382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 160/316] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:11.143181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.242940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd6c580-28bf-4263-b4e5-7bd3425666eb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.276737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da1b4fcf-4207-4cd5-8756-54e7a2d32397 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.277375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1b4fcf-4207-4cd5-8756-54e7a2d32397 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.277647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1b4fcf-4207-4cd5-8756-54e7a2d32397 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.277917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da1b4fcf-4207-4cd5-8756-54e7a2d32397 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.279065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1b4fcf-4207-4cd5-8756-54e7a2d32397 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.280505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da1b4fcf-4207-4cd5-8756-54e7a2d32397 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.282145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 157/317] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:11.288127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:11.389423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e92118e-645b-43f8-b545-69999faf3e75 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:11.424483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-631e89d9-8fe3-42a0-aa4c-1ee95d72dfa4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:11.425093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631e89d9-8fe3-42a0-aa4c-1ee95d72dfa4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.425363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631e89d9-8fe3-42a0-aa4c-1ee95d72dfa4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.425628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-631e89d9-8fe3-42a0-aa4c-1ee95d72dfa4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.426377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631e89d9-8fe3-42a0-aa4c-1ee95d72dfa4 None admin] RBAC: Adding query filter params (role.id=837dd36862a94b73aa5d4d06027541fc, scope.project.id=bb140e38668841f4a920514f6d6826f2, user.id=fa76bf5317684688a052c454c37fdfd9) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:11.426678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631e89d9-8fe3-42a0-aa4c-1ee95d72dfa4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:11.428118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-631e89d9-8fe3-42a0-aa4c-1ee95d72dfa4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:11.434757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 161/318] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:11 2022] GET /identity/v3/role_assignments?user.id=fa76bf5317684688a052c454c37fdfd9&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=837dd36862a94b73aa5d4d06027541fc => generated 608 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:11.760270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4da81a-ef21-4c6b-9b74-c729302aeae6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.760651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4da81a-ef21-4c6b-9b74-c729302aeae6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.760847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4da81a-ef21-4c6b-9b74-c729302aeae6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.762226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 158/319] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:11.767632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e82fd6-bc82-477e-b2b8-9d0c822f5dd7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.767924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e82fd6-bc82-477e-b2b8-9d0c822f5dd7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.768191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01e82fd6-bc82-477e-b2b8-9d0c822f5dd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.786773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-01e82fd6-bc82-477e-b2b8-9d0c822f5dd7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:11.787956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-01e82fd6-bc82-477e-b2b8-9d0c822f5dd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:11.826040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.831346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca824691-3450-4ad9-a98a-f5e3541ef1ed None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.831858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca824691-3450-4ad9-a98a-f5e3541ef1ed None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.832306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca824691-3450-4ad9-a98a-f5e3541ef1ed None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.833778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 159/320] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:11.839274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148f9dc-03b0-4255-a399-26820caff9e3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:11.840236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148f9dc-03b0-4255-a399-26820caff9e3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:11.840833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3148f9dc-03b0-4255-a399-26820caff9e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:11.860451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3148f9dc-03b0-4255-a399-26820caff9e3 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:11.860859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 162/321] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:11.862404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3148f9dc-03b0-4255-a399-26820caff9e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:11.878428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:11.941415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 160/322] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:12.126689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e82fd6-bc82-477e-b2b8-9d0c822f5dd7 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.196862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148f9dc-03b0-4255-a399-26820caff9e3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.235605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01e82fd6-bc82-477e-b2b8-9d0c822f5dd7 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.274162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.274801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.275075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.275342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.276598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.278202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.279981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 163/323] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:12.285331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.305346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3148f9dc-03b0-4255-a399-26820caff9e3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.328862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.345481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655fef5f-6d0b-483b-b426-94dcc16ab486 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.346162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655fef5f-6d0b-483b-b426-94dcc16ab486 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.346439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655fef5f-6d0b-483b-b426-94dcc16ab486 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.346709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655fef5f-6d0b-483b-b426-94dcc16ab486 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.358377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655fef5f-6d0b-483b-b426-94dcc16ab486 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-655fef5f-6d0b-483b-b426-94dcc16ab486 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:12.363254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:12.364320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 161/324] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/roles/reader => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:12.365949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea2940-dd11-469c-87f4-bd01b9f27de9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.366706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea2940-dd11-469c-87f4-bd01b9f27de9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.366706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eea2940-dd11-469c-87f4-bd01b9f27de9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.367517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 162/325] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:12 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:12.370698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eea2940-dd11-469c-87f4-bd01b9f27de9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.382297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.391481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dc2c42-75c3-4bea-8230-813d031dacad None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.423915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa05f2a-f424-4b42-b5f7-6ab7326efb14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.424512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa05f2a-f424-4b42-b5f7-6ab7326efb14 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.424778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa05f2a-f424-4b42-b5f7-6ab7326efb14 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.425037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa05f2a-f424-4b42-b5f7-6ab7326efb14 None admin] PATH_INFO: `/v3/groups` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.425652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa05f2a-f424-4b42-b5f7-6ab7326efb14 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.427260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa05f2a-f424-4b42-b5f7-6ab7326efb14 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.445061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 164/326] 10.222.0.61 () {60 vars in 1134 bytes} [Thu Sep 29 01:59:12 2022] POST /identity/v3/groups => generated 226 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 01:59:12.446425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eeb940-558c-4a0e-8297-2302e7170fb5 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.446792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eeb940-558c-4a0e-8297-2302e7170fb5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.447129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4eeb940-558c-4a0e-8297-2302e7170fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.467134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c4eeb940-558c-4a0e-8297-2302e7170fb5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:12.468068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c4eeb940-558c-4a0e-8297-2302e7170fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:12.487603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eea2940-dd11-469c-87f4-bd01b9f27de9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.525497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.526188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.526462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.526729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.527401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:12.527703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.529154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.545640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 163/327] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:12.549484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.556490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 165/328] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:12.626051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.661784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d5d925e-bf3e-4141-8adc-1afd881cab87 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.696501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.697160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.697499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.697744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] PATH_INFO: `/v3/users/9d8cbbd3fe224f6593cc21552efd312d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.699609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] RBAC: Authorizing `identity:get_user(user_id=9d8cbbd3fe224f6593cc21552efd312d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.701070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.702747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 164/329] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/users/9d8cbbd3fe224f6593cc21552efd312d => generated 293 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:12.708593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eeb940-558c-4a0e-8297-2302e7170fb5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.739975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:12.817320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4eeb940-558c-4a0e-8297-2302e7170fb5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:12.829103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:12.863758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:12.864523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:12.864792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:12.865058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] PATH_INFO: `/v3/system/users/9d8cbbd3fe224f6593cc21552efd312d/roles/837dd36862a94b73aa5d4d06027541fc` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:12.868412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=9d8cbbd3fe224f6593cc21552efd312d, role_id=837dd36862a94b73aa5d4d06027541fc)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:12.868412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:12.879266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 166/330] 10.222.0.61 () {58 vars in 1334 bytes} [Thu Sep 29 01:59:12 2022] PUT /identity/v3/system/users/9d8cbbd3fe224f6593cc21552efd312d/roles/837dd36862a94b73aa5d4d06027541fc => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:12.973073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9fb7d4-0daf-4567-ad7d-05a16d56456f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.012050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.012751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.013027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.013294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.023059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:13.029706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:13.034679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:13.035598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:13.036372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 165/331] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:12 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:13.042449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.146090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:13.155028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31eada66-36ba-4c94-a4c7-1ba3e63219bd None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.190737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.191352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.191621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.191886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.192545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:13.192893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.194257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.203212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 167/332] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:13 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:13.208850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.316129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d111d55c-d7df-4e90-823a-3b51dc844b2f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.350849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f35857cd-5eb7-4063-aae2-79ac57edf42b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.351540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35857cd-5eb7-4063-aae2-79ac57edf42b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.351811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35857cd-5eb7-4063-aae2-79ac57edf42b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.352124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f35857cd-5eb7-4063-aae2-79ac57edf42b None admin] PATH_INFO: `/v3/users/1e5d36533af0452382fc6776816b5b54` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.353257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35857cd-5eb7-4063-aae2-79ac57edf42b None admin] RBAC: Authorizing `identity:get_user(user_id=1e5d36533af0452382fc6776816b5b54)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.355188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f35857cd-5eb7-4063-aae2-79ac57edf42b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.357085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 166/333] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:13 2022] GET /identity/v3/users/1e5d36533af0452382fc6776816b5b54 => generated 293 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:13.362962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:13.478952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0747e761-4ec5-4ce9-ac17-526d18aaaac1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:13.520939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2ca68a-0671-4988-af30-183b362f53c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:13.521549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ca68a-0671-4988-af30-183b362f53c3 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:13.521817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ca68a-0671-4988-af30-183b362f53c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:13.522106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ca68a-0671-4988-af30-183b362f53c3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:13.522834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ca68a-0671-4988-af30-183b362f53c3 None admin] RBAC: Adding query filter params (role.id=9a454063cf7f41f1aeab078ffae345e0, user.id=1e5d36533af0452382fc6776816b5b54, scope.system=all) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:13.523134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ca68a-0671-4988-af30-183b362f53c3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:13.524642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ca68a-0671-4988-af30-183b362f53c3 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:13.537789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 168/334] 10.222.0.61 () {56 vars in 1308 bytes} [Thu Sep 29 01:59:13 2022] GET /identity/v3/role_assignments?user.id=1e5d36533af0452382fc6776816b5b54&scope.system=all&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 510 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:14.952547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdf5430-d974-4a8f-b67d-e95341fc2110 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:14.952547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdf5430-d974-4a8f-b67d-e95341fc2110 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:14.952547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcdf5430-d974-4a8f-b67d-e95341fc2110 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:14.953732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 167/335] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:14.965361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b3e07-ad84-4d63-a2c2-0246165e555b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:14.966291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b3e07-ad84-4d63-a2c2-0246165e555b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:14.966758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68b3e07-ad84-4d63-a2c2-0246165e555b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:14.993754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-e68b3e07-ad84-4d63-a2c2-0246165e555b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:14.995967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e68b3e07-ad84-4d63-a2c2-0246165e555b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:15.014782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:15.095917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 169/336] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:15.590030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addbc877-224e-4645-a4ab-166712b4c49e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.590307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addbc877-224e-4645-a4ab-166712b4c49e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.590828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addbc877-224e-4645-a4ab-166712b4c49e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.592605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 168/337] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:15.603746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb554cab-1bfd-4f3a-93ee-09b98e0b2956 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.604408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb554cab-1bfd-4f3a-93ee-09b98e0b2956 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.604869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb554cab-1bfd-4f3a-93ee-09b98e0b2956 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.630995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-bb554cab-1bfd-4f3a-93ee-09b98e0b2956 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:15.632511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bb554cab-1bfd-4f3a-93ee-09b98e0b2956 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:15.652722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addbc877-224e-4645-a4ab-166712b4c49e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:15.731439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 170/338] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:15.746998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:15.797577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addbc877-224e-4645-a4ab-166712b4c49e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:15.837584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15abf377-e155-4632-81a9-c4eb89c63ffa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:15.838381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15abf377-e155-4632-81a9-c4eb89c63ffa None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.838848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15abf377-e155-4632-81a9-c4eb89c63ffa None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.839415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15abf377-e155-4632-81a9-c4eb89c63ffa None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.850333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15abf377-e155-4632-81a9-c4eb89c63ffa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-15abf377-e155-4632-81a9-c4eb89c63ffa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:15.857531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:15.858446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:15.859232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 169/339] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:15 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:15.864757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb554cab-1bfd-4f3a-93ee-09b98e0b2956 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:15.878639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e95f8-9274-4c9a-9353-2e97d4386673 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.878913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e95f8-9274-4c9a-9353-2e97d4386673 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.879371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e95f8-9274-4c9a-9353-2e97d4386673 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.881073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 170/340] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:15.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d7e7ed-69af-4e93-808e-4d66abbd6723 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:15.887821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d7e7ed-69af-4e93-808e-4d66abbd6723 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:15.888229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d7e7ed-69af-4e93-808e-4d66abbd6723 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:15.908558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-42d7e7ed-69af-4e93-808e-4d66abbd6723 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:15.910141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-42d7e7ed-69af-4e93-808e-4d66abbd6723 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:15.998521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.031429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 171/341] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:16.033287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e573a9-f4b5-49c4-b00c-666316dac51e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.033704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e573a9-f4b5-49c4-b00c-666316dac51e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.033813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb554cab-1bfd-4f3a-93ee-09b98e0b2956 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.037401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e573a9-f4b5-49c4-b00c-666316dac51e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.040634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 172/342] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:16 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:16.046065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26c3939-af2f-48b7-996b-1b935cd0d363 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.046321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26c3939-af2f-48b7-996b-1b935cd0d363 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.046616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f26c3939-af2f-48b7-996b-1b935cd0d363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.083284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f26c3939-af2f-48b7-996b-1b935cd0d363 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:16.084671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f26c3939-af2f-48b7-996b-1b935cd0d363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:16.114745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.115383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.115665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.115929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.116607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:16.116906 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.118339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.133251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 171/343] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:16.139185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.186560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.214073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 173/344] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:16.302257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8509415-9523-4e3a-b10d-996f99837230 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.316489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26c3939-af2f-48b7-996b-1b935cd0d363 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.354779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.357324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.357611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.357879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.367006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.487046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.487666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f26c3939-af2f-48b7-996b-1b935cd0d363 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.488510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.537823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.539664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.539933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.540194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.560909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:16.567437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:16.568309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:16.569107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:16.572115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 174/345] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 256 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:16.573589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.615387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 172/346] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 476 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:59:16.616346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.631080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.658341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.696586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a32cde0-4e38-4648-b433-5d9d76a27c35 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.736202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62b7aa9c-e3c9-4751-aa70-b5d4ff710acc None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.737733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.738566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.739020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.739699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.753867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:16.759766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:16.761198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 29 01:59:16.762251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:16.765058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 175/347] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/roles/reader => generated 84 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:16.769464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.779884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.780637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.781003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.781385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.782798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:16.784460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.784658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.794127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 173/348] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:16.795048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.890176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10be9911-14b2-4132-adc9-8f5121469c78 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.899340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.911428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:16.915916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7133c5e-654b-4524-9d1c-612a085a0dc6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:16.935170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.936065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.936347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.936621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.938179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.939642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.946116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 176/349] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:16.947116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:16.962543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:16.963563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:16.964093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:16.964705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:16.965652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:16.966186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:16.967959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:16.980096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 174/350] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:16.981463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.066937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf6169e2-25c6-4e17-a17f-483e5f8431af None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.095075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da41edb9-9cbd-401d-9f9f-7da2304ba7f6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.111901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.113272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.113977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.114466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.128960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:17.135002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:17.136104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:17.136909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:17.137506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.138125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.138374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.138644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.139896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 177/351] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:17.140986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.177857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.192183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.242840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.245759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.281896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd7e837-83af-44a9-93ea-d884a8729a2e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.331564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.332569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.333048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.333499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.341507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.343437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.350169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 178/352] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.350169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.388435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 175/353] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:17.389297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.458631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.479868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.484489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9989cd-7039-4c58-bbf0-348209c43d9c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.514985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99a8293c-c28d-4334-b280-c8baa28aefef None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.529950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d4552bf-4edf-433a-87ef-8db940fb5371 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.531201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4552bf-4edf-433a-87ef-8db940fb5371 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.531475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4552bf-4edf-433a-87ef-8db940fb5371 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.531804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d4552bf-4edf-433a-87ef-8db940fb5371 None admin] PATH_INFO: `/v3/users/9d8cbbd3fe224f6593cc21552efd312d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.534743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4552bf-4edf-433a-87ef-8db940fb5371 None admin] RBAC: Authorizing `identity:get_user(user_id=9d8cbbd3fe224f6593cc21552efd312d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.536600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d4552bf-4edf-433a-87ef-8db940fb5371 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.542643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 179/354] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/users/9d8cbbd3fe224f6593cc21552efd312d => generated 293 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.542643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ef4be1-92ec-439c-b28c-08545970c837 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.542643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ef4be1-92ec-439c-b28c-08545970c837 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.542643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03ef4be1-92ec-439c-b28c-08545970c837 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.542643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 180/355] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:17 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:17.546151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ef4be1-92ec-439c-b28c-08545970c837 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.577371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.577995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.578335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.578596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.579264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.579564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.580948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.590280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 176/356] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.590998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.684061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03ef4be1-92ec-439c-b28c-08545970c837 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.717942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016f3bc4-120c-4793-b3dc-9babb063adcf None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.735698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.736309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.736749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.737025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.738093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=507e44cb9594402a9fd0ac69d80cd885) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.738406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.741173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.743959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.750364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 181/357] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 260 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.751608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.772020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee3eda7-205f-4063-bac2-6fd29a2fc888 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.773141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3eda7-205f-4063-bac2-6fd29a2fc888 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.773654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3eda7-205f-4063-bac2-6fd29a2fc888 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.774144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3eda7-205f-4063-bac2-6fd29a2fc888 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.777287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee3eda7-205f-4063-bac2-6fd29a2fc888 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.779359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee3eda7-205f-4063-bac2-6fd29a2fc888 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.781365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 177/358] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.783226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724621a-d550-480c-8899-4b9c8fc0d0f8 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.783226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724621a-d550-480c-8899-4b9c8fc0d0f8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.783226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9724621a-d550-480c-8899-4b9c8fc0d0f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.794454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:17.818323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9724621a-d550-480c-8899-4b9c8fc0d0f8 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:17.819790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9724621a-d550-480c-8899-4b9c8fc0d0f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:17.899468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257f1fcf-e7a6-4964-ad63-0e67bf50b037 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:17.923666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 178/359] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:17.924567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724621a-d550-480c-8899-4b9c8fc0d0f8 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:17.959206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:17.960363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:17.960642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:17.960916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:17.961642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] RBAC: Adding query filter params (role.id=837dd36862a94b73aa5d4d06027541fc, user.id=9d8cbbd3fe224f6593cc21552efd312d, scope.system=all) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:17.961951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:17.964916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:17.979776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 182/360] 10.222.0.61 () {56 vars in 1308 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/role_assignments?user.id=9d8cbbd3fe224f6593cc21552efd312d&scope.system=all&role.id=837dd36862a94b73aa5d4d06027541fc => generated 510 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:17.980089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.092829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.108171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9724621a-d550-480c-8899-4b9c8fc0d0f8 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.109398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50531bbb-8268-4a29-893a-79aba33f7718 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.143880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.155747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b63e414-8fc4-4e90-b9fd-04f23352b5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.156400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b63e414-8fc4-4e90-b9fd-04f23352b5c1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.156676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b63e414-8fc4-4e90-b9fd-04f23352b5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.156998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b63e414-8fc4-4e90-b9fd-04f23352b5c1 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.158221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b63e414-8fc4-4e90-b9fd-04f23352b5c1 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.159988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b63e414-8fc4-4e90-b9fd-04f23352b5c1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.161705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 183/361] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:18.163320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54b105a-48be-4da7-b7a7-672ec3bfde12 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.163692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54b105a-48be-4da7-b7a7-672ec3bfde12 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.163969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54b105a-48be-4da7-b7a7-672ec3bfde12 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.169443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 184/362] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:18.170558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54b105a-48be-4da7-b7a7-672ec3bfde12 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.174848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab20416-c222-457b-b10b-2251907a440c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.175484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab20416-c222-457b-b10b-2251907a440c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.175757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab20416-c222-457b-b10b-2251907a440c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.176022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab20416-c222-457b-b10b-2251907a440c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.283278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab20416-c222-457b-b10b-2251907a440c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.285551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab20416-c222-457b-b10b-2251907a440c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.297203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54b105a-48be-4da7-b7a7-672ec3bfde12 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.338098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-510f266f-5c97-46ef-b1d6-91206cc4254a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.338825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510f266f-5c97-46ef-b1d6-91206cc4254a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.339126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510f266f-5c97-46ef-b1d6-91206cc4254a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.339416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-510f266f-5c97-46ef-b1d6-91206cc4254a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.340152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510f266f-5c97-46ef-b1d6-91206cc4254a None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=507e44cb9594402a9fd0ac69d80cd885) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:18.340465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510f266f-5c97-46ef-b1d6-91206cc4254a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.342064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-510f266f-5c97-46ef-b1d6-91206cc4254a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.349551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 185/363] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 260 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:18.350892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f9efa-cc58-4379-a97f-b5b9da798471 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.351269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f9efa-cc58-4379-a97f-b5b9da798471 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.351601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-210f9efa-cc58-4379-a97f-b5b9da798471 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.374304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-210f9efa-cc58-4379-a97f-b5b9da798471 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:18.375546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-210f9efa-cc58-4379-a97f-b5b9da798471 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:18.392919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.421588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 179/364] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:17 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 497 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:18.422838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab20416-c222-457b-b10b-2251907a440c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.466076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.480115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 186/365] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:18.481418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f9efa-cc58-4379-a97f-b5b9da798471 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.565284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab20416-c222-457b-b10b-2251907a440c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.611171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.611967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.612240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.612506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.640880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:18.648014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:18.648913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:18.649721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:18.655580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 180/366] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:18.666433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.689137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-210f9efa-cc58-4379-a97f-b5b9da798471 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.767713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.768358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.768632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.768903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.770493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.770992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.772632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.774309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 187/367] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 293 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:18.780539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.809661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb68a31a-3d56-4969-b51a-40967af45885 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.828481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:18.885159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.886343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.886861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.887409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.888377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:18.888862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.890566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.900147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 181/368] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:18.901454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:18.940293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a65a385-e56a-4a4b-b55c-572fd7201030 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:18.976792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:18.977430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:18.977706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:18.977980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:18.979157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:18.980865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:18.987871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 188/369] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:18.987871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.021114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-437f62e9-1958-48b4-985c-fd64df563a4a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.059764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.060773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.061259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.061801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.073376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:19.078149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:19.081546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:19.082448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:19.083206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:19.085511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 182/370] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:19.086602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.091561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:19.094685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1688359b-615d-42ab-9222-297b3f1983db None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.143103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.143723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.144004 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.144570 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.215173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbd77b6-ed1b-49d3-9dea-7e78f70a6f0c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.234802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:19.236540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.255949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.256914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.257427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.257929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.259969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=bb140e38668841f4a920514f6d6826f2) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:19.260566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:19.262534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.269330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 183/371] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:19 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 260 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:19.270445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.338565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:19.358242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 189/372] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:18 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 375 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:19.364591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.410056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:19.478668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33746e28-85d7-4c1b-be7b-15c6bb332654 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.486899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7049ed25-f71d-4df3-91c6-632b14b554d5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.530327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.531123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.531402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.531673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.533428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:19.535430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.537720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 190/373] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:19 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:19.543568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.548569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.549465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.549926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.550401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.551262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:19.551745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:19.553455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.571625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 184/374] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:19 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:19.577874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.652555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:19.718225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422a32c0-a853-4c2b-a401-3c593329444c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.753368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-227e471b-bfe1-4838-9845-1897da3691f6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:19.779644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.780643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.781104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.781556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.783071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:19.786057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.788927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 191/375] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:19 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:19.795884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:19.808535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:19.822142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:19.823232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:19.823737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:19.824282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:19.949792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:19.951652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:19.984860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573e18c-0548-40c2-9535-97e03c4dea91 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.016918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.031880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.032910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.033465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.033951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.034902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=bb140e38668841f4a920514f6d6826f2) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:20.035401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.037300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.045168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 192/376] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:19 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 260 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:20.111784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 185/377] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 534 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:20.117260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.255324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543e626d-bfd3-4a02-987f-9c31e3986208 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.294601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df7234e-eecb-4890-935d-9d250aa448bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.295262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df7234e-eecb-4890-935d-9d250aa448bc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.295537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df7234e-eecb-4890-935d-9d250aa448bc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.295806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df7234e-eecb-4890-935d-9d250aa448bc None admin] PATH_INFO: `/v3/projects/c6a6000fb0914b0390fa25e08205ee14` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.297062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df7234e-eecb-4890-935d-9d250aa448bc None admin] RBAC: Authorizing `identity:get_project(project_id=c6a6000fb0914b0390fa25e08205ee14)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.298575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df7234e-eecb-4890-935d-9d250aa448bc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.300295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 193/378] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/projects/c6a6000fb0914b0390fa25e08205ee14 => generated 344 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:20.306030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.320476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5559c90d-21e6-4224-883c-f6680ef8fad8 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.320793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5559c90d-21e6-4224-883c-f6680ef8fad8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.321033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5559c90d-21e6-4224-883c-f6680ef8fad8 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.325271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 194/379] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:20 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:20.332102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2837eb-434b-469c-b51f-6734ec328523 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.332379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2837eb-434b-469c-b51f-6734ec328523 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.332666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af2837eb-434b-469c-b51f-6734ec328523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.347869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.361156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-af2837eb-434b-469c-b51f-6734ec328523 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:20.362336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-af2837eb-434b-469c-b51f-6734ec328523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:20.459647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.480705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 195/380] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:20.488192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9b19d9-f692-43e3-9022-48679fddc58b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.551307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.551968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.552317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.552585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] PATH_INFO: `/v3/groups/64a2808ef50b45af8898bab815d0f7d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.559595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] RBAC: Authorizing `identity:get_group(group_id=64a2808ef50b45af8898bab815d0f7d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.561078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.563045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 186/381] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/groups/64a2808ef50b45af8898bab815d0f7d7 => generated 229 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:20.568693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2837eb-434b-469c-b51f-6734ec328523 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.706440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af2837eb-434b-469c-b51f-6734ec328523 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:20.758246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:20.758858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:20.759131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:20.759400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:20.760141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] RBAC: Adding query filter params (group.id=64a2808ef50b45af8898bab815d0f7d7, role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=c6a6000fb0914b0390fa25e08205ee14) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:20.760486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:20.761948 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:20.770462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 196/382] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/role_assignments?group.id=64a2808ef50b45af8898bab815d0f7d7&scope.project.id=c6a6000fb0914b0390fa25e08205ee14&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 260 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:20.817398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:20.945832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:20.955729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af0c6743-1971-460a-866f-ad5d33bfedc1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.002840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2d4940-1ea9-453e-9f5a-10ea152f6f0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.003500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d4940-1ea9-453e-9f5a-10ea152f6f0f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.003779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d4940-1ea9-453e-9f5a-10ea152f6f0f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.004048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d4940-1ea9-453e-9f5a-10ea152f6f0f None admin] PATH_INFO: `/v3/roles/member` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.017655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2d4940-1ea9-453e-9f5a-10ea152f6f0f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-df2d4940-1ea9-453e-9f5a-10ea152f6f0f None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:21.023012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:21.023948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:21.025179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 187/383] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:21.030379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.110618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.179592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda34a41-2e65-4c4a-8516-76e97fdc0ad2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.196301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b6ae4-4c4c-4bae-a193-f8a2bb778a40 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.196568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b6ae4-4c4c-4bae-a193-f8a2bb778a40 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.196824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083b6ae4-4c4c-4bae-a193-f8a2bb778a40 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.198749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 188/384] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:21.214640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b967f-031f-4f0c-96d8-5997ecba8f34 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.215207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b967f-031f-4f0c-96d8-5997ecba8f34 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.215674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6b967f-031f-4f0c-96d8-5997ecba8f34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.226187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.226810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.227079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.227343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.228005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:21.228300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.230175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.250104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 197/385] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:21.251070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.255023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-4b6b967f-031f-4f0c-96d8-5997ecba8f34 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:21.256512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4b6b967f-031f-4f0c-96d8-5997ecba8f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:21.376400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 189/386] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:21.427884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29dbbdb2-0953-4728-bbf3-2eddcaeb61e5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.450342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.478714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.479448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.479720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.479992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.481152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.482614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.484470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 198/387] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:21.490142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6b967f-031f-4f0c-96d8-5997ecba8f34 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.618739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:21.648838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6b967f-031f-4f0c-96d8-5997ecba8f34 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.687629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.688273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.688544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.688826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.689932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.691411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.693327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 190/388] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:21.699241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.823900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fe63e5-51ce-48ec-a2a1-b688c910aa2e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:21.856899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:21.873194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:21.873870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:21.874168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:21.874436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:21.877796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] RBAC: Authorizing `identity:create_grant(project_id=507e44cb9594402a9fd0ac69d80cd885, group_id=5fb16c509eeb49819163b1fdbb003bb7, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:21.879512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:21.910226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 199/389] 10.222.0.61 () {58 vars in 1442 bytes} [Thu Sep 29 01:59:21 2022] PUT /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:21.998181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35977a42-69a0-47d7-b2ee-b344e32b1c68 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.042951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.043968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.044438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.044923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.059890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:22.065061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:22.066013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:22.066803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:22.069338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 191/390] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:21 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:22.075609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.125119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.206537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a6bbb3-a082-4f77-a68f-c42a1222ce56 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.248508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.249127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.249398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.249665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.250358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:22.250659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.252037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.265839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 200/391] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:22.271652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.287224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.392946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a22967f-dd5c-4a70-a94b-e3b7929f4a26 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.430511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75521c9d-d8cd-406d-917a-f79a5c27c481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.431147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75521c9d-d8cd-406d-917a-f79a5c27c481 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.431418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75521c9d-d8cd-406d-917a-f79a5c27c481 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.431685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75521c9d-d8cd-406d-917a-f79a5c27c481 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.433159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75521c9d-d8cd-406d-917a-f79a5c27c481 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.434667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75521c9d-d8cd-406d-917a-f79a5c27c481 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.436483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 192/392] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:22.442974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.505171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f58f5-7af1-411f-9cc5-8357b1b3ce3c None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.505354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f58f5-7af1-411f-9cc5-8357b1b3ce3c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.505608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3f58f5-7af1-411f-9cc5-8357b1b3ce3c None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.507301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 193/393] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:22.516297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d21fdf8-9416-4a22-bd42-14c2f7e772a0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.516626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d21fdf8-9416-4a22-bd42-14c2f7e772a0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.516914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d21fdf8-9416-4a22-bd42-14c2f7e772a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.561159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3d21fdf8-9416-4a22-bd42-14c2f7e772a0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:22.562902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3d21fdf8-9416-4a22-bd42-14c2f7e772a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:22.615882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.623121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a9d50c-cae6-4c62-8ef2-1edbdf020c4f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.679430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 194/394] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:22.683102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.684094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.684579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.685054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.686471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.688135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.690330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 201/395] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:22.696081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d21fdf8-9416-4a22-bd42-14c2f7e772a0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:22.722412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:22.822177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d21fdf8-9416-4a22-bd42-14c2f7e772a0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:22.860074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7128a2a-59df-4f70-b810-692be1744461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:22.861143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7128a2a-59df-4f70-b810-692be1744461 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:22.861620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7128a2a-59df-4f70-b810-692be1744461 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:22.862152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7128a2a-59df-4f70-b810-692be1744461 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:22.864805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7128a2a-59df-4f70-b810-692be1744461 None admin] RBAC: Authorizing `identity:create_grant(project_id=507e44cb9594402a9fd0ac69d80cd885, group_id=5fb16c509eeb49819163b1fdbb003bb7, role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:22.866657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7128a2a-59df-4f70-b810-692be1744461 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:22.889290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 195/396] 10.222.0.61 () {58 vars in 1442 bytes} [Thu Sep 29 01:59:22 2022] PUT /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/1b55a134a7b24115b2bb8691600d2619 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:22.999642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.021027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17cc33-926a-4a59-becc-d532b80dd075 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.021710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17cc33-926a-4a59-becc-d532b80dd075 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.021987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17cc33-926a-4a59-becc-d532b80dd075 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.023398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 196/397] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:23.031060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700eac62-8b50-4311-b48d-05a96406a63d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.031678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700eac62-8b50-4311-b48d-05a96406a63d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.032010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700eac62-8b50-4311-b48d-05a96406a63d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.059053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-700eac62-8b50-4311-b48d-05a96406a63d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:23.060397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-700eac62-8b50-4311-b48d-05a96406a63d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:23.143753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0af126d1-1350-4626-b3ca-eef6ba2f2354 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.165747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 197/398] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:23 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:23.166418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.204090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.204848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.205187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.205463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.217639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:23.224052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:23.224932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:23.225717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 202/399] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:22 2022] GET /identity/v3/roles/member => generated 84 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:23.231495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700eac62-8b50-4311-b48d-05a96406a63d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.332578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700eac62-8b50-4311-b48d-05a96406a63d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.377385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.378020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.378338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.378612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.379277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:23.379570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.380958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.393535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 198/400] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:23.398987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.437641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.500749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.533858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d221e47-a4f3-4a44-9a25-10bfb326f548 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.566456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.584658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.585287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.585556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.585821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.586967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.588415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.590485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 203/401] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:23.601276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.633046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7e56a1a-5106-42d1-b2a6-f4a4e9f3baa0 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.700062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.700679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.700955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.701226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.730703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:23.736081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:23.736951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:23.737746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 199/402] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:23.744662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.785048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.798283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acc7997-e5a2-4224-99c2-743370299b97 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.876254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.880064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.880345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.880619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.883572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.885031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.887153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 204/403] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:23.893958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:23.939055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa3d4fa-eb11-4e57-b99a-de664c2ed10a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:23.969639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:23.984696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f27916be-7743-4ee3-bb9e-bafc425565fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:23.985395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27916be-7743-4ee3-bb9e-bafc425565fb None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.985677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27916be-7743-4ee3-bb9e-bafc425565fb None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.985937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f27916be-7743-4ee3-bb9e-bafc425565fb None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:23.986712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27916be-7743-4ee3-bb9e-bafc425565fb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:23.987012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27916be-7743-4ee3-bb9e-bafc425565fb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:23.988500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f27916be-7743-4ee3-bb9e-bafc425565fb None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:23.997468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 200/404] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:23 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:23.998760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20842015-b2b7-4c17-b674-708a371ecff6 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:23.999104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20842015-b2b7-4c17-b674-708a371ecff6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:23.999382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20842015-b2b7-4c17-b674-708a371ecff6 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.004960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 201/405] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:24.006179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a81f0-05cd-45cf-84b9-6d1ce81afd0e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.006457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a81f0-05cd-45cf-84b9-6d1ce81afd0e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.006725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d6a81f0-05cd-45cf-84b9-6d1ce81afd0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.025797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8d6a81f0-05cd-45cf-84b9-6d1ce81afd0e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:24.027451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8d6a81f0-05cd-45cf-84b9-6d1ce81afd0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:24.037383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4daf19d3-014c-46e8-b803-81706b5ad9ba None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.074177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.074837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.075113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.075383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.077398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] RBAC: Authorizing `identity:create_grant(project_id=bb140e38668841f4a920514f6d6826f2, group_id=5fb16c509eeb49819163b1fdbb003bb7, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.078918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.098651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 205/406] 10.222.0.61 () {58 vars in 1442 bytes} [Thu Sep 29 01:59:23 2022] PUT /identity/v3/projects/bb140e38668841f4a920514f6d6826f2/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:59:24.099754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.106355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 202/407] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:24.129698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.228745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000e3f20-8298-46b1-90f8-d4741dd4d849 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.265402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.266065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.266346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.266610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.267800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.269331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.271228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 206/408] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:24.277040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a81f0-05cd-45cf-84b9-6d1ce81afd0e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.383267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d6a81f0-05cd-45cf-84b9-6d1ce81afd0e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.397734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.409378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.429073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.429722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.429994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.430278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.431952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.433803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.435761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 203/409] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:24.450038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.521001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.531127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d0297e-df31-4b7a-847e-5f72b8828ee5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.562017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d87824d-de90-4a83-b968-1258ad577d7c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.573457 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.574130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.574409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.575005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.588356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:24.595165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:24.596078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:24.596884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:24.597513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 207/410] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:24.601200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.601869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.602322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.602592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.603354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.604996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] RBAC: Authorizing `identity:create_grant(project_id=bb140e38668841f4a920514f6d6826f2, group_id=5fb16c509eeb49819163b1fdbb003bb7, role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.606826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.638639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 204/411] 10.222.0.61 () {58 vars in 1442 bytes} [Thu Sep 29 01:59:24 2022] PUT /identity/v3/projects/bb140e38668841f4a920514f6d6826f2/groups/5fb16c509eeb49819163b1fdbb003bb7/roles/1b55a134a7b24115b2bb8691600d2619 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 01:59:24.742257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0e96170-b3ec-4cd4-8d7f-e116680ef150 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.781711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e13bead-4b54-47a1-afa1-100004f3775b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.782274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13bead-4b54-47a1-afa1-100004f3775b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.782545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13bead-4b54-47a1-afa1-100004f3775b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.782813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13bead-4b54-47a1-afa1-100004f3775b None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.783495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13bead-4b54-47a1-afa1-100004f3775b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:24.783791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13bead-4b54-47a1-afa1-100004f3775b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.785225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13bead-4b54-47a1-afa1-100004f3775b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.795980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 208/412] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:24.807996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:24.869035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:24.933149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64669227-cf6d-4aac-b85b-ba0b0e6ea466 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:24.952395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b601c981-5b0c-4ea9-83ee-dcb91785cae9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.953055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b601c981-5b0c-4ea9-83ee-dcb91785cae9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.953517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b601c981-5b0c-4ea9-83ee-dcb91785cae9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.955542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 209/413] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:24 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:24.961607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a86293-041c-459d-9523-4b3c408b43c1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.962166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a86293-041c-459d-9523-4b3c408b43c1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.962644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a86293-041c-459d-9523-4b3c408b43c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.980459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:24.981103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:24.981378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:24.981645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] PATH_INFO: `/v3/groups/64a2808ef50b45af8898bab815d0f7d7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:24.982792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] RBAC: Authorizing `identity:get_group(group_id=64a2808ef50b45af8898bab815d0f7d7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:24.984258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:24.990083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 205/414] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/groups/64a2808ef50b45af8898bab815d0f7d7 => generated 229 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:24.991395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d4a86293-041c-459d-9523-4b3c408b43c1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:24.992796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d4a86293-041c-459d-9523-4b3c408b43c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:24.997331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.005531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.104394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 210/415] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:25.154598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ae1c943-bd6c-4373-85a4-db091ddfbd68 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.169444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.198745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.199802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.200078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.200347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] PATH_INFO: `/v3/projects/c6a6000fb0914b0390fa25e08205ee14` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.201478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] RBAC: Authorizing `identity:get_project(project_id=c6a6000fb0914b0390fa25e08205ee14)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:25.202969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:25.209562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 206/416] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:24 2022] GET /identity/v3/projects/c6a6000fb0914b0390fa25e08205ee14 => generated 344 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:25.211069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a86293-041c-459d-9523-4b3c408b43c1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.327827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a86293-041c-459d-9523-4b3c408b43c1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.367524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.368545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.369006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.369459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] PATH_INFO: `/v3/projects/c6a6000fb0914b0390fa25e08205ee14/groups/64a2808ef50b45af8898bab815d0f7d7/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.371763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] RBAC: Authorizing `identity:create_grant(project_id=c6a6000fb0914b0390fa25e08205ee14, group_id=64a2808ef50b45af8898bab815d0f7d7, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:25.373414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:25.400049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 211/417] 10.222.0.61 () {58 vars in 1442 bytes} [Thu Sep 29 01:59:25 2022] PUT /identity/v3/projects/c6a6000fb0914b0390fa25e08205ee14/groups/64a2808ef50b45af8898bab815d0f7d7/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:25.421326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.539476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acee9837-af39-478e-8c7a-7e6247dc997f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.585464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322ba88c-1c52-4758-9df8-e91bf833b2aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.586182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322ba88c-1c52-4758-9df8-e91bf833b2aa None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.586461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322ba88c-1c52-4758-9df8-e91bf833b2aa None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.586724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322ba88c-1c52-4758-9df8-e91bf833b2aa None admin] PATH_INFO: `/v3/roles/member` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.604235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322ba88c-1c52-4758-9df8-e91bf833b2aa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-322ba88c-1c52-4758-9df8-e91bf833b2aa None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:25.609710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:25.610797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:25.611584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 207/418] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:25.619071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.624153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.734273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700fa982-d5ad-4162-92f2-ae53a6eb6924 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.763200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c261427-51dd-46cf-b35b-64eaf902384b None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.763742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c261427-51dd-46cf-b35b-64eaf902384b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.764015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c261427-51dd-46cf-b35b-64eaf902384b None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.770144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 208/419] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:25 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:25.771634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa76483-38ec-43bf-8178-a3fefd0aa2e7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.772118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa76483-38ec-43bf-8178-a3fefd0aa2e7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.772639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa76483-38ec-43bf-8178-a3fefd0aa2e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.774730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.775345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.775619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.775885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:25.776718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:25.777186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:25.782027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:25.788289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 212/420] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:25.796747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:25.806262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6fa76483-38ec-43bf-8178-a3fefd0aa2e7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:25.808165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6fa76483-38ec-43bf-8178-a3fefd0aa2e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:25.888160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.894418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:25.932836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85e1b34c-ba7a-4b0f-b315-fcf4b00d8132 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:25.968821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:25.969844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:25.970344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:25.970809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.015859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 209/421] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 245 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:26.047974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:26.049856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:26.126481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:26.166415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 213/422] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:26.172740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa76483-38ec-43bf-8178-a3fefd0aa2e7 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.279863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fa76483-38ec-43bf-8178-a3fefd0aa2e7 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:26.330213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:26.331228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.331700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.332159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.336820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:26.337748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.339263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:26.341694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 210/423] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:26.347220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.385982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:26.484013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8666cb88-85dc-4acc-be4a-756c4a5f27e7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:26.537845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:26.537845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.537845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.537845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.538270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:26.539227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:26.541393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 211/424] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:26.549955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.560524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a996bc6-18c2-4d1f-9e01-4b328387c348 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:26.639551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:26.640233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.640508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.640782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.664679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:26.672476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:26.673360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:26.674181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 214/425] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:26.682741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:26.685715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-315b2cb2-770a-468a-89f4-2f4143f90231 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:26.687874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:26.738299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:26.752433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:26.753663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.760271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.760780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.761821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=507e44cb9594402a9fd0ac69d80cd885) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:26.762383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:26.764054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:26.774733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 212/426] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 611 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:26.838781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67c17a1d-a8c7-401f-b926-a56f2eddd526 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:26.875432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc7ebc3-1ccf-4065-ab2a-25ebdbf67612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:26.876395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7ebc3-1ccf-4065-ab2a-25ebdbf67612 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:26.876861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7ebc3-1ccf-4065-ab2a-25ebdbf67612 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:26.877342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc7ebc3-1ccf-4065-ab2a-25ebdbf67612 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:26.878257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc7ebc3-1ccf-4065-ab2a-25ebdbf67612 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:26.878761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc7ebc3-1ccf-4065-ab2a-25ebdbf67612 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:26.880377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc7ebc3-1ccf-4065-ab2a-25ebdbf67612 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:26.894186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 215/427] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:26.896496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.024860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47fae7bc-2a36-4531-9880-b4b2a423d986 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.071128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.072142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.072620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.073194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.122796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.163116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.165034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.175117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81807199-c6c8-4c7d-8fc0-574bf337e967 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.175327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81807199-c6c8-4c7d-8fc0-574bf337e967 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.175488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81807199-c6c8-4c7d-8fc0-574bf337e967 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.177513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 216/428] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:27.184903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f855c94-750e-4328-99a2-a527a0ffc2fc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.185180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f855c94-750e-4328-99a2-a527a0ffc2fc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.185449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f855c94-750e-4328-99a2-a527a0ffc2fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.215039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0f855c94-750e-4328-99a2-a527a0ffc2fc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:27.216698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0f855c94-750e-4328-99a2-a527a0ffc2fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:27.282976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 213/429] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:27.288462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.299678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 217/430] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:27.353773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.391403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc3041bf-6bf2-44f6-82bb-93f36e9d91fe None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.425830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e864c68f-00e7-4a07-bcda-4396c92ae89d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.426504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e864c68f-00e7-4a07-bcda-4396c92ae89d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.426777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e864c68f-00e7-4a07-bcda-4396c92ae89d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.427055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e864c68f-00e7-4a07-bcda-4396c92ae89d None admin] PATH_INFO: `/v3/projects/507e44cb9594402a9fd0ac69d80cd885` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.428166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e864c68f-00e7-4a07-bcda-4396c92ae89d None admin] RBAC: Authorizing `identity:get_project(project_id=507e44cb9594402a9fd0ac69d80cd885)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.429619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e864c68f-00e7-4a07-bcda-4396c92ae89d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.431583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 214/431] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/projects/507e44cb9594402a9fd0ac69d80cd885 => generated 298 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:27.437741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f855c94-750e-4328-99a2-a527a0ffc2fc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.462273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.563509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f855c94-750e-4328-99a2-a527a0ffc2fc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.600646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721be09f-1531-445d-875d-c990d8261262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.602916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710ef921-e5e7-4c06-8882-1ef8e0879dc0 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.603477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710ef921-e5e7-4c06-8882-1ef8e0879dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.603477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-710ef921-e5e7-4c06-8882-1ef8e0879dc0 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.604974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 215/432] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:27.605187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721be09f-1531-445d-875d-c990d8261262 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.605465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721be09f-1531-445d-875d-c990d8261262 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.605829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-721be09f-1531-445d-875d-c990d8261262 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.607142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721be09f-1531-445d-875d-c990d8261262 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.608589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-721be09f-1531-445d-875d-c990d8261262 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.610757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 218/433] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:27.611669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6023b86-aafe-450a-8841-181a6cf838f0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.612148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6023b86-aafe-450a-8841-181a6cf838f0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.612628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6023b86-aafe-450a-8841-181a6cf838f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.616414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721be09f-1531-445d-875d-c990d8261262 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.634587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b6023b86-aafe-450a-8841-181a6cf838f0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:27.635788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b6023b86-aafe-450a-8841-181a6cf838f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:27.721385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.725413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 216/434] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:27.726449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6023b86-aafe-450a-8841-181a6cf838f0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:27.750716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-721be09f-1531-445d-875d-c990d8261262 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.794688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.795806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.796143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.796712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.797558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=507e44cb9594402a9fd0ac69d80cd885) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:27.797840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:27.799269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:27.807084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 219/435] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=507e44cb9594402a9fd0ac69d80cd885&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 611 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:27.825315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:27.861601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6023b86-aafe-450a-8841-181a6cf838f0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:27.905468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831a0f5d-da62-416c-9d16-219d1c07c4f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:27.906165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a0f5d-da62-416c-9d16-219d1c07c4f7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:27.906444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a0f5d-da62-416c-9d16-219d1c07c4f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:27.906716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a0f5d-da62-416c-9d16-219d1c07c4f7 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:27.919591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-831a0f5d-da62-416c-9d16-219d1c07c4f7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-831a0f5d-da62-416c-9d16-219d1c07c4f7 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:27.924875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:27.925848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:27.926693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 217/436] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:27.934644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.033023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce39843-7121-4106-9f1e-19c34f0f793c None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.033297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce39843-7121-4106-9f1e-19c34f0f793c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.033777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce39843-7121-4106-9f1e-19c34f0f793c None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.037094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 218/437] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 01:59:28.041723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d8d263-a85b-4960-94ab-3ceb6985c58f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.042362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d8d263-a85b-4960-94ab-3ceb6985c58f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.043485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d8d263-a85b-4960-94ab-3ceb6985c58f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.067717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-85d8d263-a85b-4960-94ab-3ceb6985c58f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:28.069237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-85d8d263-a85b-4960-94ab-3ceb6985c58f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:28.083734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46a6a3f-b76a-4778-8655-af66cdc2220d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.122433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.123862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.124328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.124605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.125296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] RBAC: Adding query filter params (name=member) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.125589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.127069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.136851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 220/438] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 01:59:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:28.137788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.149642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 219/439] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:28.150576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d8d263-a85b-4960-94ab-3ceb6985c58f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.158200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.220126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.285786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d619969-7d8e-4313-873d-b16b81d82658 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.287181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d8d263-a85b-4960-94ab-3ceb6985c58f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.325686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.325889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.326228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.328111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.332463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.333260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.333545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.333866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.337846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:28.343162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:28.344037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:28.344839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 221/440] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:28.349937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.409621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.411366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.423668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.453435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b221efc7-4900-4a7c-b2a4-0fcc14c7279f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.464319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.486671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.487270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.487545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.487816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.488471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.488770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.490226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.498036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 222/441] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:28.498822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.510063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 220/442] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:28.510063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.602034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fd73a84-152f-47b2-8307-8f2658443510 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.607901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a8fbbe-892e-42ed-bcaa-1d0086623cde None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.635131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.635772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.636052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.636323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.638179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.641535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.642165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.642437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.642697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.647356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:28.652740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:28.653607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:28.654435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 223/443] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/roles/admin => generated 83 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:28.655097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.670468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.714581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.716043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.770029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c737bfb5-4182-4808-9e4a-1a7205880b4f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.801896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.802551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.802819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.803081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.804171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.805697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.807263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 224/444] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:28.808548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.813547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 221/445] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:28.814379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.867282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.870862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:28.910480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad22b859-744a-47a6-9d4e-24bcfe4024cb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.913410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b63e5e-94cc-4ccd-959d-444baa3681c4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:28.943786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.944381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.944656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.944920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.945573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:28.945873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.946901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:28.947287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.947532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:28.947804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:28.948074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:28.949231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:28.950694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:28.952267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 222/446] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:28.954971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:28.955979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 225/447] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:28.958090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.054122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12759179-f888-4230-88f1-d559d11df1e1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.058986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98329188-56ed-49aa-a3bf-1d8feaa211cd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.078617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.084318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.090792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.091478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.091740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.092004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] PATH_INFO: `/v3/projects/bb140e38668841f4a920514f6d6826f2` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.093079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] RBAC: Authorizing `identity:get_project(project_id=bb140e38668841f4a920514f6d6826f2)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.094749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.095482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.096075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.096365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.096500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 223/448] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/projects/bb140e38668841f4a920514f6d6826f2 => generated 302 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:29.097164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.101126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.101888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=9a454063cf7f41f1aeab078ffae345e0, scope.project.id=bb140e38668841f4a920514f6d6826f2) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:29.102230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.103653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.109558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 226/449] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:28 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=9a454063cf7f41f1aeab078ffae345e0 => generated 611 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:29.110355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.208381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef8e6ab-5a97-4c40-ab93-ed8a67b102d9 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.238942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6aba1c7-4e96-440e-8dc4-e2e9c426ad13 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.252816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.253461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.253717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.253980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.282541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.284307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.284889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.285169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] PATH_INFO: `/v3/groups/5fb16c509eeb49819163b1fdbb003bb7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.287006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] RBAC: Authorizing `identity:get_group(group_id=5fb16c509eeb49819163b1fdbb003bb7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.288540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.290511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 227/450] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/groups/5fb16c509eeb49819163b1fdbb003bb7 => generated 226 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:29.296522 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.322221 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.339528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.341025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.364433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.415747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6d3b47-7c2e-41fb-ad05-5b10d11be005 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.449367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 224/451] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:29.455184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.463660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.465836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.465836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.465836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.465836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] RBAC: Adding query filter params (group.id=5fb16c509eeb49819163b1fdbb003bb7, role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=bb140e38668841f4a920514f6d6826f2) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:29.466152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.467678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.474279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 228/452] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/role_assignments?group.id=5fb16c509eeb49819163b1fdbb003bb7&scope.project.id=bb140e38668841f4a920514f6d6826f2&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 611 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:29.560252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.570364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-587b934a-0fb2-4c3c-b159-4fa817c29a3b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.607410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.608056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.608628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.608628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] PATH_INFO: `/v3/projects/c6a6000fb0914b0390fa25e08205ee14` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.609806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] RBAC: Authorizing `identity:get_project(project_id=c6a6000fb0914b0390fa25e08205ee14)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.611421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.613100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 225/453] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/projects/c6a6000fb0914b0390fa25e08205ee14 => generated 344 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:29.618926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.723467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8917b0d8-7353-40f2-9f13-ff2d1874f5ab None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.743812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:29.758510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e96ee0-7a96-4905-94de-d1de12aa4fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.759135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e96ee0-7a96-4905-94de-d1de12aa4fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.759406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e96ee0-7a96-4905-94de-d1de12aa4fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.759674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e96ee0-7a96-4905-94de-d1de12aa4fa5 None admin] PATH_INFO: `/v3/groups/64a2808ef50b45af8898bab815d0f7d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.760781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e96ee0-7a96-4905-94de-d1de12aa4fa5 None admin] RBAC: Authorizing `identity:get_group(group_id=64a2808ef50b45af8898bab815d0f7d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.762954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e96ee0-7a96-4905-94de-d1de12aa4fa5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.764698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 229/454] 10.222.0.61 () {56 vars in 1180 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/groups/64a2808ef50b45af8898bab815d0f7d7 => generated 229 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:29.771783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:29.871425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6293c47-b842-48c3-bfe6-2fa45e0dde4f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:29.903652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f579778f-5210-4b76-93ea-0078855f9f8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:29.904252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f579778f-5210-4b76-93ea-0078855f9f8b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:29.904553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f579778f-5210-4b76-93ea-0078855f9f8b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:29.904813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f579778f-5210-4b76-93ea-0078855f9f8b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:29.905527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f579778f-5210-4b76-93ea-0078855f9f8b None admin] RBAC: Adding query filter params (group.id=64a2808ef50b45af8898bab815d0f7d7, role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=c6a6000fb0914b0390fa25e08205ee14) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:29.905827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f579778f-5210-4b76-93ea-0078855f9f8b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:29.907304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f579778f-5210-4b76-93ea-0078855f9f8b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:29.912989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 226/455] 10.222.0.61 () {56 vars in 1376 bytes} [Thu Sep 29 01:59:29 2022] GET /identity/v3/role_assignments?group.id=64a2808ef50b45af8898bab815d0f7d7&scope.project.id=c6a6000fb0914b0390fa25e08205ee14&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 611 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:32.620205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da7b63-85cf-4235-ac37-c2356fc39213 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.620890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da7b63-85cf-4235-ac37-c2356fc39213 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.621001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5da7b63-85cf-4235-ac37-c2356fc39213 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.622646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 230/456] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:32.627732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eb2b4f-c83b-42b3-9d7a-168599c984f1 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.628507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eb2b4f-c83b-42b3-9d7a-168599c984f1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.628965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84eb2b4f-c83b-42b3-9d7a-168599c984f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.655035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-84eb2b4f-c83b-42b3-9d7a-168599c984f1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:32.656262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-84eb2b4f-c83b-42b3-9d7a-168599c984f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:32.664584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03e11ac-8394-4a8f-91a5-6787d6c6aed9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.665124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03e11ac-8394-4a8f-91a5-6787d6c6aed9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.665585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d03e11ac-8394-4a8f-91a5-6787d6c6aed9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.667485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 231/457] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:32 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:32.672901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0521a-04db-4835-9818-609a0c0913d1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:32.673459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0521a-04db-4835-9818-609a0c0913d1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:32.673917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a0521a-04db-4835-9818-609a0c0913d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:32.674950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:32.714027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-28a0521a-04db-4835-9818-609a0c0913d1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:32.714027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-28a0521a-04db-4835-9818-609a0c0913d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:32.776680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 227/458] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:32.776895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 232/459] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:33.050978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84eb2b4f-c83b-42b3-9d7a-168599c984f1 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.087579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e954079-e0d3-4b92-b8ac-dd45b452656f None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.088234 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e954079-e0d3-4b92-b8ac-dd45b452656f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.088785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e954079-e0d3-4b92-b8ac-dd45b452656f None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.090898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 233/460] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:33.103244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3615e51c-121a-4457-9a93-b7340eb33c8d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.103882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3615e51c-121a-4457-9a93-b7340eb33c8d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.104383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3615e51c-121a-4457-9a93-b7340eb33c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.135268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3615e51c-121a-4457-9a93-b7340eb33c8d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:33.136482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3615e51c-121a-4457-9a93-b7340eb33c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:33.188041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84eb2b4f-c83b-42b3-9d7a-168599c984f1 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.223310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.224218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.224625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.225035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.227758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 234/461] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:33.228771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3615e51c-121a-4457-9a93-b7340eb33c8d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.243918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.245493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:33.271411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:33.272613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:33.273677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:33.273677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:33.273677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:33.273677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 228/462] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:33.279090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.286859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.291714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.383514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3615e51c-121a-4457-9a93-b7340eb33c8d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.433225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94259309-bda8-46aa-9b5b-e14c93660b1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.433761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306a436-21ed-4108-9965-20e87cff065a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.434208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94259309-bda8-46aa-9b5b-e14c93660b1b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.434479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94259309-bda8-46aa-9b5b-e14c93660b1b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.434745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94259309-bda8-46aa-9b5b-e14c93660b1b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.447988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94259309-bda8-46aa-9b5b-e14c93660b1b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.450293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94259309-bda8-46aa-9b5b-e14c93660b1b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-94259309-bda8-46aa-9b5b-e14c93660b1b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:33.469325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:33.470311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:33.471201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:33.471201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:33.471201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:33.471201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 235/463] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:33.471999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab40d68-e2ba-4068-b85e-0d9ad90b2071 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.472275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab40d68-e2ba-4068-b85e-0d9ad90b2071 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.472546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab40d68-e2ba-4068-b85e-0d9ad90b2071 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.477917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 236/464] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:33.478398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df96ed6-8cb2-46df-b58d-6e78918e8d39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.478398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df96ed6-8cb2-46df-b58d-6e78918e8d39 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.478398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df96ed6-8cb2-46df-b58d-6e78918e8d39 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.478631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab40d68-e2ba-4068-b85e-0d9ad90b2071 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.478699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df96ed6-8cb2-46df-b58d-6e78918e8d39 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.478699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df96ed6-8cb2-46df-b58d-6e78918e8d39 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:33.478699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df96ed6-8cb2-46df-b58d-6e78918e8d39 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.479341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df96ed6-8cb2-46df-b58d-6e78918e8d39 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.492727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 229/465] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:33.494129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ec4d0-b499-4802-90a4-fb08d85431e5 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.494411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ec4d0-b499-4802-90a4-fb08d85431e5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.494677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0ec4d0-b499-4802-90a4-fb08d85431e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.513825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2c0ec4d0-b499-4802-90a4-fb08d85431e5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:33.515071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2c0ec4d0-b499-4802-90a4-fb08d85431e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:33.588406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.590325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.594585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab40d68-e2ba-4068-b85e-0d9ad90b2071 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.600540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 230/466] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:33.601234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ec4d0-b499-4802-90a4-fb08d85431e5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.635609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.636999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.638247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.638682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.639483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:33.639942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.641533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.667428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 237/467] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:33.667428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.710895 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0ec4d0-b499-4802-90a4-fb08d85431e5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.748214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.748828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.749269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.749679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.750999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.752663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.778748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54648bea-6546-47c1-8063-ca3b0f714405 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.786502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 231/468] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:33 2022] POST /identity/v3/users => generated 246 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:33.789501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.854174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.854888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.855163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.855432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.881445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.883413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:33.903912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:33.905044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:33.905967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:33.905967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:33.905967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:33.906169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.908214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 238/469] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:33.909573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:33.930382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:33.956458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841aafca-c955-46ed-896c-f460357df7d0 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:33.995324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:33.996280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:33.996654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:33.997019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:33.997811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:33.999426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.026160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 232/470] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:33 2022] POST /identity/v3/users => generated 248 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:34.027218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:34.052044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-939004c1-7523-4546-9f57-20450ab410d0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:34.099175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ce0d82-06b3-4286-ab79-e52a0cff3996 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:34.099827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ce0d82-06b3-4286-ab79-e52a0cff3996 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.100154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ce0d82-06b3-4286-ab79-e52a0cff3996 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.100428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ce0d82-06b3-4286-ab79-e52a0cff3996 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.115884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ce0d82-06b3-4286-ab79-e52a0cff3996 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:34.117396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1ce0d82-06b3-4286-ab79-e52a0cff3996 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b1ce0d82-06b3-4286-ab79-e52a0cff3996 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:34.133906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:34.134903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:34.136252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:34.136252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:34.136252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:34.136252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 239/471] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:34.136862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8a76b-078a-4ab2-80ab-a9b42070b666 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.137144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8a76b-078a-4ab2-80ab-a9b42070b666 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.137416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa8a76b-078a-4ab2-80ab-a9b42070b666 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.139069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 240/472] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:34.141457 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8a76b-078a-4ab2-80ab-a9b42070b666 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:34.185803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-522f63ab-772f-4a9f-9b96-1bd09f9a9674 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:34.195789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:34.229738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:34.235983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0711f154-34fe-4a91-9687-f6077820f088 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:34.236666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711f154-34fe-4a91-9687-f6077820f088 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.236939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711f154-34fe-4a91-9687-f6077820f088 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.237204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0711f154-34fe-4a91-9687-f6077820f088 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.239829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0711f154-34fe-4a91-9687-f6077820f088 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:34.239829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0711f154-34fe-4a91-9687-f6077820f088 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:34.239829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0711f154-34fe-4a91-9687-f6077820f088 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.256772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 233/473] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:34.257933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba214e7c-9c84-4ab4-8930-e45f8e697a16 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.258232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba214e7c-9c84-4ab4-8930-e45f8e697a16 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.258495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba214e7c-9c84-4ab4-8930-e45f8e697a16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.279377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa8a76b-078a-4ab2-80ab-a9b42070b666 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:34.280508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ba214e7c-9c84-4ab4-8930-e45f8e697a16 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:34.281941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ba214e7c-9c84-4ab4-8930-e45f8e697a16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:34.321964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:34.322902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.323542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.324027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.324692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:34.325096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:34.326562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.340370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 241/474] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:34.341508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:34.380775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 234/475] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:34.386031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba214e7c-9c84-4ab4-8930-e45f8e697a16 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:34.514844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:34.521507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba214e7c-9c84-4ab4-8930-e45f8e697a16 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:34.538034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-715f8e59-bcae-44b4-9c6b-b9249ae99a0b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:34.585668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:34.595361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:34.595989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.596264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.596534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.597200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:34.598697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.617296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:34.617916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.618239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.618509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.619163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:34.621341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.631327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 235/476] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:34 2022] POST /identity/v3/users => generated 249 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:34.660055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 242/477] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:34 2022] POST /identity/v3/users => generated 248 bytes in 319 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:34.698459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:34.842480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2d50d8-e4df-40fe-b943-eebb62dfa36e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:34.882012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:34.882651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:34.882921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:34.883187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:34.883869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:34.885270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:34.896600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.sql.core [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 29 01:59:34.896600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 29 01:59:34.896600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [parameters: {'id': '14a143506dd14ed89b8d4c7a7ac45432', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 29 01:59:34.896600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81683) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result.read() Sep 29 01:59:34.914425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return ref.to_dict() Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:59:34.915391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application yield resource Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application next(self.gen) Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.session.rollback() Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.force_reraise() Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise self.value Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application session.commit() Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 29 01:59:34.916281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._prepare_impl() Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.session.flush() Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._flush(objects) Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application compat.raise_( Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application flush_context.execute() Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rec.execute(self) Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 29 01:59:34.918143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 01:59:34.919215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application conn.query(q) Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result.read() Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 01:59:34.920628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [parameters: {'id': '14a143506dd14ed89b8d4c7a7ac45432', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:34.926286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 29 01:59:34.927282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:34.928232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 236/478] 10.222.0.61 () {60 vars in 1131 bytes} [Thu Sep 29 01:59:34 2022] POST /identity/v3/roles => generated 144 bytes in 218 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:34.929567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:35.054154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e11d9b6-497c-4fcf-880f-d85fdf6f861d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:35.091263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c5e551-f7ee-440d-841c-75f99c09f9c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:35.091896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c5e551-f7ee-440d-841c-75f99c09f9c3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:35.092167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c5e551-f7ee-440d-841c-75f99c09f9c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:35.092432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c5e551-f7ee-440d-841c-75f99c09f9c3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:35.103771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c5e551-f7ee-440d-841c-75f99c09f9c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a8c5e551-f7ee-440d-841c-75f99c09f9c3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:35.109163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:35.110097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:35.111144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 243/479] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 01:59:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:35.116229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:35.145917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:35.279672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea758066-9576-4b19-b520-c948f22bdec5 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:35.331661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e6bc71a-7423-415f-9db3-a3ec99f43f38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:35.332727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6bc71a-7423-415f-9db3-a3ec99f43f38 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:35.333169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6bc71a-7423-415f-9db3-a3ec99f43f38 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:35.333596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e6bc71a-7423-415f-9db3-a3ec99f43f38 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:35.334500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6bc71a-7423-415f-9db3-a3ec99f43f38 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:35.334966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6bc71a-7423-415f-9db3-a3ec99f43f38 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:35.336674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e6bc71a-7423-415f-9db3-a3ec99f43f38 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:35.346113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 237/480] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:36.425792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea032d-bf30-4ac4-8d4b-b606c222a679 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.426347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea032d-bf30-4ac4-8d4b-b606c222a679 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.426347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ea032d-bf30-4ac4-8d4b-b606c222a679 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.427994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 244/481] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:36.434265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10062cea-4169-45de-9838-41152b05ca9c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.434688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10062cea-4169-45de-9838-41152b05ca9c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.434859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10062cea-4169-45de-9838-41152b05ca9c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.459027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-10062cea-4169-45de-9838-41152b05ca9c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:36.460223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-10062cea-4169-45de-9838-41152b05ca9c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:36.562270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 238/482] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:36.746550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9162f2-22fc-4fdb-a15d-7e9d4d2f8f1e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.746724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9162f2-22fc-4fdb-a15d-7e9d4d2f8f1e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.746998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9162f2-22fc-4fdb-a15d-7e9d4d2f8f1e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.748577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 245/483] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:36.753742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc4c026-f9a8-4146-a4fa-7b1a2974c89d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:36.754032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc4c026-f9a8-4146-a4fa-7b1a2974c89d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:36.754301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc4c026-f9a8-4146-a4fa-7b1a2974c89d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:36.781796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cdc4c026-f9a8-4146-a4fa-7b1a2974c89d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:36.782995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cdc4c026-f9a8-4146-a4fa-7b1a2974c89d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:36.784972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:36.861720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 239/484] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:36.955936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9162f2-22fc-4fdb-a15d-7e9d4d2f8f1e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.014557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.086695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9162f2-22fc-4fdb-a15d-7e9d4d2f8f1e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.132023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.132661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.132934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.133202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.147360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:37.152145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:37.154016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:37.154832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 246/485] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:36 2022] GET /identity/v3/roles/service => generated 85 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:37.160624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc4c026-f9a8-4146-a4fa-7b1a2974c89d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.283201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.321060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc4c026-f9a8-4146-a4fa-7b1a2974c89d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.388623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75d17b12-98e1-49e5-ad6e-0c97dee446c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.389243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d17b12-98e1-49e5-ad6e-0c97dee446c4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.389519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d17b12-98e1-49e5-ad6e-0c97dee446c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.389795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75d17b12-98e1-49e5-ad6e-0c97dee446c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.390492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d17b12-98e1-49e5-ad6e-0c97dee446c4 None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.390795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d17b12-98e1-49e5-ad6e-0c97dee446c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.392201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75d17b12-98e1-49e5-ad6e-0c97dee446c4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.404730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.411412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 240/486] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:37.412914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8974ef-83ed-4e78-a46c-bc4af0f10c85 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.413245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8974ef-83ed-4e78-a46c-bc4af0f10c85 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.413520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da8974ef-83ed-4e78-a46c-bc4af0f10c85 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.419869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 241/487] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:37 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:37.420546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8974ef-83ed-4e78-a46c-bc4af0f10c85 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.437211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.442124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9b7887b-35e7-40bb-9a85-7fb0a8d96725 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.482898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdc7ab9-3d46-4654-bf12-726c6af3b9b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.484055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc7ab9-3d46-4654-bf12-726c6af3b9b8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.484545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc7ab9-3d46-4654-bf12-726c6af3b9b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.485015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdc7ab9-3d46-4654-bf12-726c6af3b9b8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.499790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bdc7ab9-3d46-4654-bf12-726c6af3b9b8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6bdc7ab9-3d46-4654-bf12-726c6af3b9b8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:37.506035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:37.506952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:37.507865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:37.516455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 247/488] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:37.517974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4e55d-d3f1-4e96-b128-981b00b6b6a0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.518876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4e55d-d3f1-4e96-b128-981b00b6b6a0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.519296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c4e55d-d3f1-4e96-b128-981b00b6b6a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.546350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e4c4e55d-d3f1-4e96-b128-981b00b6b6a0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:37.548032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e4c4e55d-d3f1-4e96-b128-981b00b6b6a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:37.549184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da8974ef-83ed-4e78-a46c-bc4af0f10c85 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.598371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.599012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.599284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.599556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.611703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.613169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:37.627757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:37.628857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:37.629721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:37.629721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:37.629721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:37.629721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 242/489] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:37.630128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.631596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 248/490] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:37.632983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff390b25-f5c4-4041-b18c-2d5a39e4af4b None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.633382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff390b25-f5c4-4041-b18c-2d5a39e4af4b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.633781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff390b25-f5c4-4041-b18c-2d5a39e4af4b None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.635134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 249/491] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:37.640049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff390b25-f5c4-4041-b18c-2d5a39e4af4b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.708626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.738502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:37.751511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff390b25-f5c4-4041-b18c-2d5a39e4af4b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.774908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce0f002-0d58-4247-a0df-35305a653567 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.794436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0724ca42-ca93-4ad9-bac0-d7a581741894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.795248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0724ca42-ca93-4ad9-bac0-d7a581741894 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.795685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0724ca42-ca93-4ad9-bac0-d7a581741894 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.796043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0724ca42-ca93-4ad9-bac0-d7a581741894 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.796846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0724ca42-ca93-4ad9-bac0-d7a581741894 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.797180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0724ca42-ca93-4ad9-bac0-d7a581741894 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.798802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0724ca42-ca93-4ad9-bac0-d7a581741894 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.815970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:37.816585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.817057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.817162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 250/492] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:37.818119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.818866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:37.818980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27736bb-6456-4510-8118-26713f9e5634 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:37.819172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:37.819400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27736bb-6456-4510-8118-26713f9e5634 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:37.819873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27736bb-6456-4510-8118-26713f9e5634 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:37.820614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:37.830570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 243/493] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:37.832493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.846877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c27736bb-6456-4510-8118-26713f9e5634 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:37.848313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c27736bb-6456-4510-8118-26713f9e5634 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:37.929431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 251/494] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:37.942022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27736bb-6456-4510-8118-26713f9e5634 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:37.961456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf443dc3-5ee8-4dde-885b-73a86baf8c98 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:37.976688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.001812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.002672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.003041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.003416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.013990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.015752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:38.030324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:38.031291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:38.032159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 244/495] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/users/nova => generated 82 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:38.034151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.037315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.058310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27736bb-6456-4510-8118-26713f9e5634 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.097586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.098524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.098939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.099345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.112317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.113815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:38.130852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:38.131789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:38.132623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:38.132623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:38.132623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:38.132623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 252/496] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:38.132842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.152353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4193c4-2281-4405-b809-2e2a6599fce5 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.186265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b22bcaa0-e92e-4b4d-9eb1-8809ed7617c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.186932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bcaa0-e92e-4b4d-9eb1-8809ed7617c8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.187212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bcaa0-e92e-4b4d-9eb1-8809ed7617c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.187510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b22bcaa0-e92e-4b4d-9eb1-8809ed7617c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.197542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b22bcaa0-e92e-4b4d-9eb1-8809ed7617c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-b22bcaa0-e92e-4b4d-9eb1-8809ed7617c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:38.202939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:38.203864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:38.204646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 245/497] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:38.205374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8756582b-8e61-44c3-9701-1e9a202257a0 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.205584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8756582b-8e61-44c3-9701-1e9a202257a0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.205831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8756582b-8e61-44c3-9701-1e9a202257a0 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.207119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 246/498] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:38.207804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8756582b-8e61-44c3-9701-1e9a202257a0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.242520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e5590a9-ea76-438d-b75c-2d53f5deeb36 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.244721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.271559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.277266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.277878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.278187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.278456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.279180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:38.279477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.280909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.313351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8756582b-8e61-44c3-9701-1e9a202257a0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.316846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 253/499] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.317677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.349000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.349611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.349886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.350190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.350835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:38.351161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.352556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.364326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 247/500] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.365109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.419331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878f41a1-36e8-46ea-b85c-37b3cb925e1c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.452820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e81257-1771-4ac5-9aa0-7bf53a23203a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.453430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e81257-1771-4ac5-9aa0-7bf53a23203a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.453712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e81257-1771-4ac5-9aa0-7bf53a23203a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.453968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e81257-1771-4ac5-9aa0-7bf53a23203a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.457776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.464199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e81257-1771-4ac5-9aa0-7bf53a23203a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:38.464648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdc2e5ba-f059-48f3-b705-fe54bdc8f04a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a5e81257-1771-4ac5-9aa0-7bf53a23203a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:38.469001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:38.470088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:38.470889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 254/501] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:38.471560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2197a26a-5e9a-4b61-9fac-f966227e4aa1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.472015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2197a26a-5e9a-4b61-9fac-f966227e4aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.472015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2197a26a-5e9a-4b61-9fac-f966227e4aa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.490272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2197a26a-5e9a-4b61-9fac-f966227e4aa1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:38.491458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2197a26a-5e9a-4b61-9fac-f966227e4aa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:38.500511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.501121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.501391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.501656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.502336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:38.502628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.503986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.512216 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 248/502] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.513000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.539096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.561607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 255/503] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:38.562678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2197a26a-5e9a-4b61-9fac-f966227e4aa1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.629726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-565937d3-b65d-4a2b-a68d-e585c50cde08 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.663858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.664473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.664745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.665010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.671761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2197a26a-5e9a-4b61-9fac-f966227e4aa1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.690424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.708820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.709460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.709736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.710314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.717966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.719446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:38.731860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:38.732726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:38.733617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 256/504] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/users/glance => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:38.734196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.740473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.742239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.780569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.845458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-924b7426-3ec1-4bc6-bef0-a060c121ae94 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.846419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 249/505] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:38.847265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.878731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.879333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.879608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.879926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.880532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:38.880831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.882240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:38.890104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 257/506] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:38.890890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:38.906593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.951277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ec4b51-ab92-46b4-b356-fb21fd4d7f98 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.986142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:38.986774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:38.987052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:38.987320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:38.989381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779873bf-6ec1-4c39-b6e7-ee4e9b8b92c3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:38.990924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:38.997557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:38.999061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:39.013475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:39.014436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:39.015288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:39.015288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.015288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:39.015288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 250/507] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:39.015503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.024771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.025382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.025654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.026017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.026749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:39.027048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.028467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.045427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 258/508] 10.222.0.61 () {56 vars in 1137 bytes} [Thu Sep 29 01:59:38 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:39.046250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.118702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182e62eb-1eba-48e9-813c-e5b824abb317 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.127557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.148700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6769178-006f-4874-b57e-a6c281d1909b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.154281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.154974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.155274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.155577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.166372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.168015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:39.182153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:39.183271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:39.184118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:39.184118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.184118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:39.184288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.184288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.184439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 251/509] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:39.184503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.184503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.184627 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.196923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.198437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:39.213434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:39.214480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:39.215315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:39.215315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.215315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:39.215315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 259/510] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:39.215732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.230961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.290245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ac2582-7fba-4de7-bcbe-0d80d19de665 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.316572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f96626f2-2fda-4900-b127-3779208e84da None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.323790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.324424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.324700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.324974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.337211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.339166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.348242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.348850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.349127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.349425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.350095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:39.350399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.351773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:39.352898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:39.353760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:39.354957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:39.354957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.354957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:39.354957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 252/511] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:39.354957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.363805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 260/512] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:39.364608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.369793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.433241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.456335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bb0187-0be9-4eee-ba29-f2bf4eb3f676 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.465356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-157f0581-34d8-4fac-bce2-610ad7df12ed None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.488079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.488676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.488948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.489215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.499846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.500452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.500730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.501005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.501646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:39.501947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.503366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.514740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 261/513] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:39.515533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.561599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.563147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.576543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.615796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bb2237-c4e1-464c-b84a-5708b415c502 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.645547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.651434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.652046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.652385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.652656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.653328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:39.653657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.655073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.658456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 253/514] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:39.659409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.672422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 262/515] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:39.672422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.754665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-563fb3ec-3a2f-4e3f-9813-20f4525dd82b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.765836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bb7162-4468-4968-82de-9405b6094408 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.767226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.789199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.789798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.790104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.790370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.791001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:39.791316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.792845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.799921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.800542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.800820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.801084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.808621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 254/516] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:39.808621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.809712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.811207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:39.823998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:39.824890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:39.825767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 263/517] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/users/neutron => generated 85 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:39.826362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.877888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:39.905448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf3979a-c759-46cd-aac7-1e048770bc60 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.923744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52f71bdf-1841-428a-b5d8-c54a893b62e7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:39.939866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.940481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.940760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.941093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.941733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.943245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.958856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:39.959491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:39.959768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:39.960041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:39.966820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 255/518] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:39 2022] POST /identity/v3/users => generated 247 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:39.967642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:39.971454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:39.972968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:39.989420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:39.990494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:39.996471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:39.996471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:39.996471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:39.996471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 264/519] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:39.996471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.023574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.106440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4918ba0c-7039-46cb-b4ac-8bda2db4e0d4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.112494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98539fec-2e74-48fc-b1ab-c99d78419c01 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.115180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.148263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.148908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.149191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.149306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.149501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.149882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.150185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.150502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.159828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.160262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.161297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.161734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:40.173495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:40.174428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:40.176546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:40.177089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:40.178263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:40.180484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:40.181119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 256/520] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/projects/service => generated 88 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:40.181119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.181241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 265/521] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:39 2022] GET /identity/v3/users/cinder => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:40.181241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.278449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.287574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c786e4e-0d87-4882-839e-2d8b2cadf30c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.291011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cbd8ba-a841-4ced-91b3-96d06fe949aa None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.323534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.324141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.324439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.324673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.325308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:40.325604 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.326526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.327049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.327143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.327407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.327680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.328413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:40.328717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.330190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.344308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 266/522] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:40.344308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.370185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.371012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 257/523] 10.222.0.61 () {56 vars in 1139 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:40.371956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.455225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12de17ea-6a20-471e-b6c0-16de4aa8efa5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.479917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b5ad83c-6a74-4c16-a504-5d7ad1c454f8 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.490755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.491366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.491640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.491910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.492638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:40.492978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.494623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.513440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 267/524] 10.222.0.61 () {56 vars in 1137 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:40.514215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.517056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.518564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.518841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.519110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.519814 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:40.520116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.521571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.539840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 258/525] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:40.540608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.550536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.606497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.627600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b99530-d7a4-4f64-a777-3c416c2c3e7e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.645437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40863c09-e430-4328-b4db-89613e0c165e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.667413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.668304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.668717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.668989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.676741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.678212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.681125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.682260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.682629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.682904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:40.690310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:40.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:40.691961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 268/526] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:40.692504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.760511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.762129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.770798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.799376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7f0cd87-5ae3-4df3-b09e-32b4ba7ae96f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.832718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:40.833317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:40.833587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:40.833849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:40.846512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:40.872408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 259/527] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:40.872408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:40.917019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:40.918539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:40.975770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b445fb20-9480-4844-9ae2-86b9cf65cfed None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:40.985075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.011359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.011958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.012225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.012488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.013198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=bc807d5f532b4e4c9953ba5e7eb878bb) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.013498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.014964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.020510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 260/528] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/role_assignments?user.id=bc807d5f532b4e4c9953ba5e7eb878bb&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 259 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:41.022910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.023503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 269/529] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:41.024414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.074413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.146986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fcc4af-654d-4105-a551-0c2a225d5ebc None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.166166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4878b27-ca23-4ba7-a89d-34339b2afa3a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.186505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.187126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.187403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.187672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.188432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.189174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.190301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.208459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 261/530] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:41.209807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.213082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.214178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.215517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.215997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.228950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.230971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:41.250281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:41.251263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:41.252131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:41.252131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:41.252131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:41.254108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 270/531] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:41.254597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.254975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.328151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3a0f25-c972-48b0-ab9d-21b0a38e7926 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.368386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.369239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.369625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.369846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.370008 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.376126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5764fcca-c4e1-4db8-8bb1-f6fcdc8bdad6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.381449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.383131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:41.398263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:41.399195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:41.400029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:41.400029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:41.400029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:41.400029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 262/532] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:41.401118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.414284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.415181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.415648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.416129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.417347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=d6cfd5241a0147d9bfe0b7addb3e1851) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.417831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.419485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.434177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 271/533] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/role_assignments?user.id=d6cfd5241a0147d9bfe0b7addb3e1851&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 259 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:41.435538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.507905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36783bf8-e7f5-4f24-ab7d-034381892172 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.511207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.574130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.574806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.575088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.575362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.576022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.576324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.577742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.602724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 263/534] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:41.607681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.644445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720d5528-ad00-42d7-98fb-76a9c1ac258a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.713200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.713818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.714122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.714393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.715048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:41.715339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.717391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.733303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 272/535] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:41.738753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.762213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.772613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395f5911-7ef4-4d40-8026-d1e02ec12de6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.826407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.827295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.827708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.829865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.847182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.848680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:41.876823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:41.877898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:41.883039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 264/536] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:41.892664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12770995-893a-461b-bec6-e8376bb44a43 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:41.896352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:41.908299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:41.931852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:41.932923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:41.933472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:41.933956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:41.941481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:41.943338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:41.956538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:41.957417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:41.958274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:41.960611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 273/537] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:41.987024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.054517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98a047e-d134-41a3-8e6d-9736266bd1c7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.102410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.126445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.127133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.127404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.127682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.128391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.128504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c21de278-ab23-479b-bd06-f43e476476b3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.128698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.130192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.156368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 265/538] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:42.164740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.165356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.165629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.165936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.167101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.167239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.167640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.169215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.189976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 274/539] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:42.199042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.259130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.300673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f52606-555b-4e69-b876-bddd08190a14 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.327538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bd1b0c0-490f-4ed7-9e5f-5b25ce938836 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.336532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e404eda3-3583-41c4-93e0-45342a7ed05d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.337204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e404eda3-3583-41c4-93e0-45342a7ed05d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.337486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e404eda3-3583-41c4-93e0-45342a7ed05d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.337811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e404eda3-3583-41c4-93e0-45342a7ed05d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.338683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e404eda3-3583-41c4-93e0-45342a7ed05d None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=cc2f041fe7194a3da199517b4cdbd265) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.338989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e404eda3-3583-41c4-93e0-45342a7ed05d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.340495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e404eda3-3583-41c4-93e0-45342a7ed05d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.346646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 266/540] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/role_assignments?user.id=cc2f041fe7194a3da199517b4cdbd265&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 259 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:42.347904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822ec354-5aa9-4973-bb1b-6b61c91bf6d8 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.348185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822ec354-5aa9-4973-bb1b-6b61c91bf6d8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.348456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822ec354-5aa9-4973-bb1b-6b61c91bf6d8 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.349658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 267/541] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:42 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:42.355837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8394352c-de93-4feb-a61e-4f2a233c62a1 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.356150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8394352c-de93-4feb-a61e-4f2a233c62a1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.356384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8394352c-de93-4feb-a61e-4f2a233c62a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.375759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.376472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.376750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.377027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.377839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=36c4d127aac743baaeb86157dbbeb689) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:42.378111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:42.379966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:42.387105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 275/542] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/role_assignments?user.id=36c4d127aac743baaeb86157dbbeb689&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:42.387420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8394352c-de93-4feb-a61e-4f2a233c62a1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:42.388486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8394352c-de93-4feb-a61e-4f2a233c62a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:42.467732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 268/543] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:42.741724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:42.745244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:42.850842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baf62576-bb45-4858-b102-298945f2c9e3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:42.900078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:42.901267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:42.901742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:42.902250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:42.919153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:42.925009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:42.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:42.926702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:42.929012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 276/544] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/roles/service => generated 85 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:42.939159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8394352c-de93-4feb-a61e-4f2a233c62a1 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.020231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:43.077931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8394352c-de93-4feb-a61e-4f2a233c62a1 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:43.124249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d658601-454a-473f-9967-8d8bed645078 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:43.124857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d658601-454a-473f-9967-8d8bed645078 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.125130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d658601-454a-473f-9967-8d8bed645078 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.125393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d658601-454a-473f-9967-8d8bed645078 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.126072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d658601-454a-473f-9967-8d8bed645078 None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:43.126373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d658601-454a-473f-9967-8d8bed645078 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:43.127754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d658601-454a-473f-9967-8d8bed645078 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:43.141520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 269/545] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:42 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:43.147227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.223214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:43.277269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6385e9d2-7661-4e60-bcad-361f770d436b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:43.324611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:43.325305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.325620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.325890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.346427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:43.347905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:43.365632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc76b829-a3b2-4618-abfc-9138e628ae89 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.365962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc76b829-a3b2-4618-abfc-9138e628ae89 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.366198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc76b829-a3b2-4618-abfc-9138e628ae89 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.367529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 270/546] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:43.373335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb593b52-ad09-4602-a4b1-9808520c2882 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.373615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb593b52-ad09-4602-a4b1-9808520c2882 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.373888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb593b52-ad09-4602-a4b1-9808520c2882 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:43.374324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:43.375168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:43.376002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:43.376002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:43.376002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:43.376002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 277/547] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:43.381015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.406322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-fb593b52-ad09-4602-a4b1-9808520c2882 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:43.407509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-fb593b52-ad09-4602-a4b1-9808520c2882 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:43.496153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f600f56b-9ee3-42d7-bf8d-24299de9ab72 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:43.510842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:43.551100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:43.551729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.552001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.552266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.553682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:43.554229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:43.555623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:43.574751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 271/548] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:43.585460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 278/549] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:43.591971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb593b52-ad09-4602-a4b1-9808520c2882 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.663451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:43.737483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb593b52-ad09-4602-a4b1-9808520c2882 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:43.785001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:43.785646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:43.785920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:43.786212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:43.799676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:43.801087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:43.816447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:43.817896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:43.819015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 272/550] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:43 2022] GET /identity/v3/users/swift => generated 83 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:43.825761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.883283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:43.960044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f958660f-c5bf-4c35-8a0c-b0950055f34c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.026318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.026931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.027200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.027464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.028192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:44.028492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.030061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.073764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 279/551] 10.222.0.61 () {56 vars in 1135 bytes} [Thu Sep 29 01:59:43 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:44.079313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.105057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2fd255-0525-432c-b13a-2369128eed7d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.124167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.143144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.176781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48dd256-eeef-4698-badb-0bf5ec7ef22a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.177440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48dd256-eeef-4698-badb-0bf5ec7ef22a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.177733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48dd256-eeef-4698-badb-0bf5ec7ef22a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.178024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48dd256-eeef-4698-badb-0bf5ec7ef22a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.189362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48dd256-eeef-4698-badb-0bf5ec7ef22a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a48dd256-eeef-4698-badb-0bf5ec7ef22a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:44.194983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:44.196245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:44.197374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 273/552] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:43 2022] GET /identity/v3/roles/service => generated 85 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:44.198230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dfdc81-1a30-494d-9249-1d5db8f57fb8 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.198230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dfdc81-1a30-494d-9249-1d5db8f57fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.198396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5dfdc81-1a30-494d-9249-1d5db8f57fb8 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.203815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 274/553] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:44.205038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7dce0-5abe-4caf-90fb-60d611a3a4d2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.205324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7dce0-5abe-4caf-90fb-60d611a3a4d2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.205594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69d7dce0-5abe-4caf-90fb-60d611a3a4d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.224366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-69d7dce0-5abe-4caf-90fb-60d611a3a4d2 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:44.225681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-69d7dce0-5abe-4caf-90fb-60d611a3a4d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:44.242055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3050eacc-8bd5-4b77-9579-71f5d64f8028 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.283817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.284694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.284812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.285052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.307255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 275/554] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:44.308860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d7dce0-5abe-4caf-90fb-60d611a3a4d2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.402114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.403622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.421181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69d7dce0-5abe-4caf-90fb-60d611a3a4d2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.423036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.460312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.461280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.461898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.462687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.463703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:44.464197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.465761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.474896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 276/555] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:44.475414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.481970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.612633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e4c923-cb01-469d-bccc-49687f725dbb None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.620018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 280/556] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 541 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:44.625964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.687293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.688308 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.690397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.690703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.703450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.705058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:44.724960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:44.725943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:44.726824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:44.726824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:44.726824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:44.731441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 277/557] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:44.732324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.753206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.774149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bebdad2-4ea2-4f7e-9929-1b3e4359f080 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.812682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:44.818898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.820134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.820600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.822088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.834823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:44.837398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:44.847846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a105b01c-a5ac-4bdc-b567-20da46b75275 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:44.856538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:44.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:44.858845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:44.858845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:44.858845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:44.861004 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 281/558] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 235 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:44.862271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.884422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:44.885552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:44.886030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:44.886162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:44.897122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:44.903073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:44.903979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:44.904897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:44.909163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 278/559] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:44.910051 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:44.989865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d583fc26-2ecc-44a8-8ec8-a03c5197a447 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:44.995235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.027540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff83f538-1414-4970-99b6-a71ceeefd440 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.036925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.038286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.038773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.039655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.040807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:45.041474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.043221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.059318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 282/560] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.060473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.070820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.071438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.071778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.072833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.073556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:45.073933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.075805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.093958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 279/561] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.093958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.124047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.186504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3418d3-d0fe-4c71-b03b-698fc3b95ca5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.210505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17fdf92-00a9-435c-960b-a351337e4c31 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.222467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34fd1d86-12a2-4dd6-8b2f-5a85e0cdbca6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.223070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fd1d86-12a2-4dd6-8b2f-5a85e0cdbca6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.223342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fd1d86-12a2-4dd6-8b2f-5a85e0cdbca6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.223605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34fd1d86-12a2-4dd6-8b2f-5a85e0cdbca6 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.224263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fd1d86-12a2-4dd6-8b2f-5a85e0cdbca6 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:45.224551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fd1d86-12a2-4dd6-8b2f-5a85e0cdbca6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.225914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34fd1d86-12a2-4dd6-8b2f-5a85e0cdbca6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.233783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 283/562] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.234970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629a026-844a-44d5-8e84-0ad893955d22 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.235280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629a026-844a-44d5-8e84-0ad893955d22 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.235649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7629a026-844a-44d5-8e84-0ad893955d22 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.237081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 284/563] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:45.237785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7629a026-844a-44d5-8e84-0ad893955d22 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.240751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.250323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ff772a-c286-4f2e-9084-c5dd4af3c234 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.251079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff772a-c286-4f2e-9084-c5dd4af3c234 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.251220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff772a-c286-4f2e-9084-c5dd4af3c234 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.251492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ff772a-c286-4f2e-9084-c5dd4af3c234 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.259219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ff772a-c286-4f2e-9084-c5dd4af3c234 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.260673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ff772a-c286-4f2e-9084-c5dd4af3c234 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-07ff772a-c286-4f2e-9084-c5dd4af3c234 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:45.272017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:45.273022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:45.273976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 280/564] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:45.274622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740585a6-50d4-46ee-b19c-f37e2f2c7109 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.274622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740585a6-50d4-46ee-b19c-f37e2f2c7109 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.274760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740585a6-50d4-46ee-b19c-f37e2f2c7109 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.276057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 281/565] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:45 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:45.276746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740585a6-50d4-46ee-b19c-f37e2f2c7109 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.344374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7629a026-844a-44d5-8e84-0ad893955d22 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.351136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.378324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740585a6-50d4-46ee-b19c-f37e2f2c7109 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.378739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd675f9-6ddb-4cc3-b561-d23734e828e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.379351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd675f9-6ddb-4cc3-b561-d23734e828e0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.379623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd675f9-6ddb-4cc3-b561-d23734e828e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.379891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd675f9-6ddb-4cc3-b561-d23734e828e0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.387739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd675f9-6ddb-4cc3-b561-d23734e828e0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.389196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd675f9-6ddb-4cc3-b561-d23734e828e0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-9cd675f9-6ddb-4cc3-b561-d23734e828e0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:45.400420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:45.401429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:45.402384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 285/566] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:45.402914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c4d39-6f3b-4d04-8ffd-b947bc63d3db None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.403185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c4d39-6f3b-4d04-8ffd-b947bc63d3db None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.403452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c4d39-6f3b-4d04-8ffd-b947bc63d3db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.415360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.416125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.416398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.416662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.422052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-323c4d39-6f3b-4d04-8ffd-b947bc63d3db None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:45.423248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-323c4d39-6f3b-4d04-8ffd-b947bc63d3db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:45.427858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.429294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:45.444316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:45.445418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:45.446686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:45.446686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:45.446686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:45.446686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 282/567] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:45.446686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.497626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.498113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 286/568] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:45.499351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29590f4-a50b-47dd-97ef-b9a58fb2ef05 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.499627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29590f4-a50b-47dd-97ef-b9a58fb2ef05 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.499874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e29590f4-a50b-47dd-97ef-b9a58fb2ef05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.518887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e29590f4-a50b-47dd-97ef-b9a58fb2ef05 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:45.520070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e29590f4-a50b-47dd-97ef-b9a58fb2ef05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:45.563851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc83c8a2-f59f-4975-89b6-8220f6c560a0 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.602035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 287/569] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:45.602035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29590f4-a50b-47dd-97ef-b9a58fb2ef05 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.609504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.610755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.611202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.612036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.613201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:45.613683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.616050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.627997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.644083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 283/570] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.644083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.711578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e29590f4-a50b-47dd-97ef-b9a58fb2ef05 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.750906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.751454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.751727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.751998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.752688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:45.752987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.754647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.767380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cbe650-5b86-4789-9a46-e12db7d047ac None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.767951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 288/571] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.768645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.768942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.803837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.804444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.804715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.804980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.805637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:45.805939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.807756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.819027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 284/572] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:45.819780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.867394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:45.881847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a042909-71ba-4484-9c25-4d6aba882276 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.914742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.915606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.915869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.916131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.922576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a30eeb-9c65-4ec9-94d0-bda18f692d35 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:45.927094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:45.928552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:45.942798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:45.943855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:45.944720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:45.944720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:45.944720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:45.944720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 289/573] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:45.944720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:45.968560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:45.969194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:45.969472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:45.969745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:45.979992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:45.985134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:45.986243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:45.987095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 285/574] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:45.987769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.012614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c365d09-79c5-4adf-8ed9-9d59c83daf2d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.084377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.084979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.085256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.085524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.086280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=34b928e9fe314d0fb61122b7a558c1b7) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.086376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2a0e6-52e1-4767-99e3-c2d25a54b8df None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.086584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.088170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.093574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 290/575] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/role_assignments?user.id=34b928e9fe314d0fb61122b7a558c1b7&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.094385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.096877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.149113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.151136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.151861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.152280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.164866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.167936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:46.181568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.182755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 286/576] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:45 2022] GET /identity/v3/users/glance => generated 84 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.183721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.217016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb47635-bda6-4e51-8b20-5399f9766043 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.256016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.257176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.257516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.257838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.269259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:46.273834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:46.275364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.277170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 291/577] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.278226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.299528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d0fd56-a501-4d7c-bc23-8213eba37883 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.334460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281526f-db38-470d-a785-a47aada8345c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.335221 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281526f-db38-470d-a785-a47aada8345c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.335592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281526f-db38-470d-a785-a47aada8345c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.336292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f281526f-db38-470d-a785-a47aada8345c None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.336661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f281526f-db38-470d-a785-a47aada8345c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.337036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f281526f-db38-470d-a785-a47aada8345c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.338640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f281526f-db38-470d-a785-a47aada8345c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.346226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.351986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 287/578] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.353384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281526f-db38-470d-a785-a47aada8345c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.392000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ddb5996-ed63-4b2d-a6d4-37feab346c58 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.428028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.428635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.428910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.429237 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.429920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.430240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.432047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.441349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 292/579] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.442285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.464884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f281526f-db38-470d-a785-a47aada8345c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.498685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.499109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.499379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.499648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.500387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.500643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.502109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.517470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.521923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 288/580] 10.222.0.61 () {56 vars in 1137 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.522943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.571797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d572824-9bcb-49de-b2f2-2bce1bb42995 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.596486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.608927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.609594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.609871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.610173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.610850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:46.611150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.612516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.625665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 293/581] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:46.625665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.638433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac99e3ce-6e65-46ef-b85c-7695efe4df8e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.673050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.673679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.673980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.674251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.682425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.683886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:46.695231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.696152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 289/582] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.697079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.734958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b61838cb-9243-428d-9484-6b54d92d068a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.764330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.770520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.771157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.771434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.771705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.782767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.784155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:46.799344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.800533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:46.801617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:46.801617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.801617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:46.801617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 294/583] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.801617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.812541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea8b4da-d379-4510-85a5-7d2905377929 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.846211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:46.847470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.848342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.848799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.849136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.859624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.861091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:46.875323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.876326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:46.877349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:46.877349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.877349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:46.877349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 290/584] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.878804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.914132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbed05e2-5425-41d8-ba3b-9eb66b9e9257 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:46.950693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:46.951347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:46.951629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:46.951896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:46.964621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:46.966131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:46.980658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:46.981612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:46.982547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:46.982547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:46.982547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:46.982547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 295/585] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:46.982547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:46.994906 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df788d06-3d21-4414-81e7-ec170b0ebda8 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.009429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.032728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.033381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.033600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.033875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.034577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.034881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.036349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.053210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 291/586] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.053210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.085937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.097037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8201ab95-d53d-4635-810c-c590ba2d7406 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.131549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.132144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.132417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.132682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.133320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.133619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.135081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.146568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 296/587] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.147349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.161037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c67390c-3490-4345-b531-929bc39a714c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.199363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.199976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.200255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.200525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.201178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.201478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.202897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.218984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 292/588] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.218984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.248240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.267190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8f01e2-4a7d-498b-968e-1ba709e6f72c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.309286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.310081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.310356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.310622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.311429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.312042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.313415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.321276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cdd5be-e99f-4aac-9b51-ac669794af11 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.324875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 297/589] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.325682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.338595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.365118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.365781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.366088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.366360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/bc807d5f532b4e4c9953ba5e7eb878bb/roles/5c49b35bed6a4d3290bab54c9fa73b9e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.380999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=bc807d5f532b4e4c9953ba5e7eb878bb, role_id=5c49b35bed6a4d3290bab54c9fa73b9e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.382513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.397317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 293/590] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:47 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/bc807d5f532b4e4c9953ba5e7eb878bb/roles/5c49b35bed6a4d3290bab54c9fa73b9e => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:47.411057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.453364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ea196fb-b431-4e20-81bf-5ef0aae5a92b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.507632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.508277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.508547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.508809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.521816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.522648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.542147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:47.545275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:47.551235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 298/591] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/users/cinder => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:47.552210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.580263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb66d53-8339-4d87-bbc1-3af257d3b938 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.623639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.624285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.624564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.624833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.632149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.633608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:47.645280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:47.646282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:47.647170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:47.651172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.651698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 294/592] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:47.652248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.684276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c99928-6e0d-4af5-9105-8ad7ee1fc94b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.719368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.720047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.720318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.720578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.728973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.730548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:47.742158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:47.744395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 299/593] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/users/neutron => generated 85 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:47.746639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.756600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4851ded9-d386-4c39-adcf-de1fdf07209f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.792488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.793165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.793445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.793715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.794482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.794786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.796262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.812434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.816771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 295/594] 10.222.0.61 () {56 vars in 1137 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.818578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.850485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-676cb52d-a53a-4c2c-b86b-b631bc7a3a73 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.880615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:47.887111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.888218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.888677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.889109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.889983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.890453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.892022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.905848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 300/595] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.907014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:47.926730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b6400c8-25a9-462c-96c0-e3cb0b24c277 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:47.963531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:47.963812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:47.964151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:47.964428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:47.965160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:47.965459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:47.966996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:47.985691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 296/596] 10.222.0.61 () {56 vars in 1139 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:47.986549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.013825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68cb2086-3d71-4c33-bc68-50a38127420b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.045922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.050561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.051427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.051836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.052304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.064749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.066443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:48.083600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:48.084756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:48.085818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:48.085818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:48.085818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:48.088194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 301/597] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:48.089422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.090773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d209d645-bc35-4588-bcfe-3217a7e104e4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.127638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.128249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.128526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.128798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/d6cfd5241a0147d9bfe0b7addb3e1851/roles/5c49b35bed6a4d3290bab54c9fa73b9e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.142836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.146126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=d6cfd5241a0147d9bfe0b7addb3e1851, role_id=5c49b35bed6a4d3290bab54c9fa73b9e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.147669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.170298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 297/598] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:47 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/d6cfd5241a0147d9bfe0b7addb3e1851/roles/5c49b35bed6a4d3290bab54c9fa73b9e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:48.171459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.209657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbbf42a-2fd9-4b91-b220-184be85ab931 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.249389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-996ae555-ea22-4d24-8b00-fb88a2387848 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.250090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ae555-ea22-4d24-8b00-fb88a2387848 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.250364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ae555-ea22-4d24-8b00-fb88a2387848 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.250633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-996ae555-ea22-4d24-8b00-fb88a2387848 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.262934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996ae555-ea22-4d24-8b00-fb88a2387848 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.264421 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-996ae555-ea22-4d24-8b00-fb88a2387848 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-996ae555-ea22-4d24-8b00-fb88a2387848 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:48.280220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:48.281416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:48.282464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:48.282464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:48.282464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:48.282464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 302/599] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:48.282866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fb29c8-dbe9-4dcc-9878-000cd058f6d3 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.283141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fb29c8-dbe9-4dcc-9878-000cd058f6d3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.283408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66fb29c8-dbe9-4dcc-9878-000cd058f6d3 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.289052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 303/600] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:48 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:48.289966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fb29c8-dbe9-4dcc-9878-000cd058f6d3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.335557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7730a72f-abb6-4559-ba5b-ee3ea5a8990b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.360639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.386610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e837d46a-cf32-429d-90bb-419c9690d04a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.387477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837d46a-cf32-429d-90bb-419c9690d04a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.387890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837d46a-cf32-429d-90bb-419c9690d04a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.388389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e837d46a-cf32-429d-90bb-419c9690d04a None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.389295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e837d46a-cf32-429d-90bb-419c9690d04a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:48.389736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e837d46a-cf32-429d-90bb-419c9690d04a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.391350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e837d46a-cf32-429d-90bb-419c9690d04a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.398640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.407284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 298/601] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:48.407748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66fb29c8-dbe9-4dcc-9878-000cd058f6d3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.410043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405fbbf8-6edc-4777-a74e-3c998afe233a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.410877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405fbbf8-6edc-4777-a74e-3c998afe233a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.411140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405fbbf8-6edc-4777-a74e-3c998afe233a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.433931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-405fbbf8-6edc-4777-a74e-3c998afe233a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:48.435550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-405fbbf8-6edc-4777-a74e-3c998afe233a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:48.445153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.445994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.446541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.446896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.447698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:48.448195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.449721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.465327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 304/602] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:48.466314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.526026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 299/603] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:48.526026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405fbbf8-6edc-4777-a74e-3c998afe233a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.623847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed1182c0-58d8-4706-a961-3852308d5647 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.663810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.664356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.664649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.664900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.679488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.682315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:48.703133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:48.704147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:48.705070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:48.708289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 305/604] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:48.709813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.728131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405fbbf8-6edc-4777-a74e-3c998afe233a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.730305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.752826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:48.793948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.794615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.794885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.795145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.803423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.804892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:48.819227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:48.820202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:48.821095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 300/605] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:48.821612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.863471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ab660f-d462-4462-ab47-fc8ad4e26db8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.903949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.904585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.904858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.905118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:48.905798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:48.906121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:48.907554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:48.921117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 306/606] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:48.921788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:48.941090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ece0a09-37a5-4467-a9fa-4b129c649732 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:48.987660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:48.988871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:48.989573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:48.990072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.004465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:49.008162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:49.011478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:49.012419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:49.013304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:49.020584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 301/607] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:49.020584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:49.045971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf356f61-6504-433e-81c0-6c5c4ed05998 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.050057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:49.085678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.086615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.087182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.087770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.088607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:49.089190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.091437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.110151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 307/608] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:49.111811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:49.164170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac843fa9-c681-466a-8166-20c2d1199e17 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.203662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.204643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.205081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.205492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/36c4d127aac743baaeb86157dbbeb689/roles/5c49b35bed6a4d3290bab54c9fa73b9e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.222585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=36c4d127aac743baaeb86157dbbeb689, role_id=5c49b35bed6a4d3290bab54c9fa73b9e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.224282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.233800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-166c137b-7b78-4b2f-8b7d-641c23f87cfc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.241791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 302/609] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:49 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/36c4d127aac743baaeb86157dbbeb689/roles/5c49b35bed6a4d3290bab54c9fa73b9e => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:49.243038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:49.272492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.273298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.273683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.274316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.275368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:49.276230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.278029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.280044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:49.289039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 308/610] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:49 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:49.294986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:49.329113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:49.377260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bd0de8-e02c-4a43-b296-0176ce295e2b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.464007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98de860e-ea3b-4b49-ac52-d57fdefd061e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.478804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.479505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.479780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.480049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/cc2f041fe7194a3da199517b4cdbd265/roles/5c49b35bed6a4d3290bab54c9fa73b9e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.504055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7a9db7-2810-4bda-8764-50e598f2e1c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.504840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7a9db7-2810-4bda-8764-50e598f2e1c7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.505394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7a9db7-2810-4bda-8764-50e598f2e1c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.505587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7a9db7-2810-4bda-8764-50e598f2e1c7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.508896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=cc2f041fe7194a3da199517b4cdbd265, role_id=5c49b35bed6a4d3290bab54c9fa73b9e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.510764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.518487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7a9db7-2810-4bda-8764-50e598f2e1c7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.520077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7a9db7-2810-4bda-8764-50e598f2e1c7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.530257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 303/611] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:49 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/cc2f041fe7194a3da199517b4cdbd265/roles/5c49b35bed6a4d3290bab54c9fa73b9e => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8a7a9db7-2810-4bda-8764-50e598f2e1c7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:49.543055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:49.544251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:49.545354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:49.545354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:49.545354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:49.546826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 309/612] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:49.554202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:49.716184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68412f52-0f0a-4aa6-89ae-2927e47e2942 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.740708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:49.741921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ad4f5-0d80-4f1c-94de-9791574a41cf None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.742399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ad4f5-0d80-4f1c-94de-9791574a41cf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.742825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240ad4f5-0d80-4f1c-94de-9791574a41cf None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.744718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 310/613] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:49.751854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75376a0-6941-4585-bc61-bc9e09b306de None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.752226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75376a0-6941-4585-bc61-bc9e09b306de None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.752516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75376a0-6941-4585-bc61-bc9e09b306de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.763400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.764012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.764285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.764550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:49.765188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:49.765483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:49.766912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:49.779873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 304/614] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:49.787078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f75376a0-6941-4585-bc61-bc9e09b306de None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:49.788282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f75376a0-6941-4585-bc61-bc9e09b306de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:49.791289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:49.839358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:49.865850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 311/615] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:49.944818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba984d2f-d8a2-49b4-9470-dc42bc083fb9 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:49.997563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:49.998156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:49.998430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:49.998695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] PATH_INFO: `/v3/users/swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.012269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:50.013744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:50.029208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:50.030283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:50.031214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:50.031745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 305/616] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:49 2022] GET /identity/v3/users/swift => generated 83 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:50.036047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75376a0-6941-4585-bc61-bc9e09b306de None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.165398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.189945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75376a0-6941-4585-bc61-bc9e09b306de None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:50.243138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:50.262985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:50.263721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:50.264173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:50.264553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.265299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:50.265599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:50.267566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:50.301865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 312/617] 10.222.0.61 () {56 vars in 1135 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:50.308282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.351111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:50.375947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe60bc1d-3f56-4e3e-a4f1-a6c3d86f5e6e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:50.449783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:50.451180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:50.451804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:50.453295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.474960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00eb612d-faca-4bc8-bf3e-946bfb55b68d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:50.475694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:50.484579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:50.485703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:50.486784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:50.494372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 306/618] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 329 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:50.498305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.549256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:50.554775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:50.555053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:50.555327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.583213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:50.584695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:50.619432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:50.620595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:50.622429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:50.622429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:50.622429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:50.622429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 313/619] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:50.629103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.637307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:50.642850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ec39e03-5d19-4b5b-a431-10d026aa38d5 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:50.681063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:50.681726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:50.682023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:50.682311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.683487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:50.683798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:50.685188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:50.694691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 307/620] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:50.701390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.837872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3c0fcf4-8534-447e-8eb6-96946fbc7307 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:50.852338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:50.866189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d349a5fe-8dd8-42e4-b288-edd9b813f61a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:50.885593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:50.886365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:50.886650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:50.886924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.900716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:50.902495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:50.902965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-291c8366-7ecd-48eb-a8ff-3d666b951d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:50.903580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291c8366-7ecd-48eb-a8ff-3d666b951d0e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:50.903857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291c8366-7ecd-48eb-a8ff-3d666b951d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:50.904124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-291c8366-7ecd-48eb-a8ff-3d666b951d0e None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.904828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291c8366-7ecd-48eb-a8ff-3d666b951d0e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:50.905132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291c8366-7ecd-48eb-a8ff-3d666b951d0e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:50.907025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-291c8366-7ecd-48eb-a8ff-3d666b951d0e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:50.918865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:50.919843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:50.921961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:50.921961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:50.921961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:50.922195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 314/621] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:50.928193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59bd06a-6eac-42ac-b468-f30e92ce5f2b None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:50.928193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59bd06a-6eac-42ac-b468-f30e92ce5f2b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:50.928193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59bd06a-6eac-42ac-b468-f30e92ce5f2b None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:50.928690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 315/623] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:50 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:50.928934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 308/623] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:50.929360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59bd06a-6eac-42ac-b468-f30e92ce5f2b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.930979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:50.994375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.042107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9189448-a3d8-4ac2-a261-4075964235f0 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.043820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59bd06a-6eac-42ac-b468-f30e92ce5f2b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.086409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.086409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.086409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.086409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.086881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ef23640-85f4-42f2-b59c-ef0c665928e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.086881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef23640-85f4-42f2-b59c-ef0c665928e2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.086881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef23640-85f4-42f2-b59c-ef0c665928e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.086881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ef23640-85f4-42f2-b59c-ef0c665928e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.087566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef23640-85f4-42f2-b59c-ef0c665928e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:51.087566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef23640-85f4-42f2-b59c-ef0c665928e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.089397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ef23640-85f4-42f2-b59c-ef0c665928e2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.092960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.094531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.103467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 309/624] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:51.104871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78791323-e069-4d05-9c7f-e374458db676 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.105155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78791323-e069-4d05-9c7f-e374458db676 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.105441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78791323-e069-4d05-9c7f-e374458db676 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:51.108282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:51.109187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:51.110280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 316/625] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:51.111886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.124533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.134344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-78791323-e069-4d05-9c7f-e374458db676 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:51.136102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-78791323-e069-4d05-9c7f-e374458db676 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:51.225036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 310/626] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:51.226495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78791323-e069-4d05-9c7f-e374458db676 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.237475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe9a09a-8361-430d-90f8-3dd1aa9da060 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.246169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.291280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.293686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.294175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.294589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.309942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.311969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:51.326060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:51.327230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:51.328422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:51.330650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 317/627] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:51 2022] GET /identity/v3/users/nova => generated 82 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:51.337861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.397978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78791323-e069-4d05-9c7f-e374458db676 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.459840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.466559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.467141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.467492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.467866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.470102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:51.471051 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.472775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.494250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 311/628] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:51.500655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.503750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44f3c211-a472-45e8-9b19-113ad27d2807 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.545132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.546043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.546473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.546908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.548275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:51.548742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.550727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.573624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 318/629] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:51.574717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.574892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.627565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd97de1e-3122-4e37-b03f-2589ce99f986 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.669827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.670750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.671187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.671602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.682778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:51.688208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:51.689397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:51.690524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:51.692907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 312/630] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:51.693848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.701551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d25c96-eaae-473a-890b-ad6bc3b3b53c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.733701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.745246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.745952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.746316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.746689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/34b928e9fe314d0fb61122b7a558c1b7/roles/5c49b35bed6a4d3290bab54c9fa73b9e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:51.764420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=34b928e9fe314d0fb61122b7a558c1b7, role_id=5c49b35bed6a4d3290bab54c9fa73b9e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:51.765955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:51.786094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 319/631] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 01:59:51 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/34b928e9fe314d0fb61122b7a558c1b7/roles/5c49b35bed6a4d3290bab54c9fa73b9e => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 01:59:51.787314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:51.827797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19913ef9-9ba6-4ded-bf5a-538bb7d9a2a4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:51.885407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:51.907901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bd08091-4094-4fd0-b720-5f31a6bd3899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:51.908833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd08091-4094-4fd0-b720-5f31a6bd3899 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:51.909272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd08091-4094-4fd0-b720-5f31a6bd3899 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:51.909696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bd08091-4094-4fd0-b720-5f31a6bd3899 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.009723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70877121-b7df-4b47-9b94-a2a38ecbbf30 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.011283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd08091-4094-4fd0-b720-5f31a6bd3899 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.013542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bd08091-4094-4fd0-b720-5f31a6bd3899 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.053790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1a7829-d37e-4075-b66e-86857a8617a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.056006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a7829-d37e-4075-b66e-86857a8617a5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.056006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a7829-d37e-4075-b66e-86857a8617a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.056006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1a7829-d37e-4075-b66e-86857a8617a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.056422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1a7829-d37e-4075-b66e-86857a8617a5 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:52.056560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1a7829-d37e-4075-b66e-86857a8617a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.057940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1a7829-d37e-4075-b66e-86857a8617a5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.066726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 320/632] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:52.069016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd290c51-e217-4044-8c96-07531e83e4a6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.069016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd290c51-e217-4044-8c96-07531e83e4a6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.069016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd290c51-e217-4044-8c96-07531e83e4a6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.074141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 321/633] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:52 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:52.079067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd290c51-e217-4044-8c96-07531e83e4a6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.123753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.142029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 313/634] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 444 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:52.142029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c8300-8b45-4c9b-ba8f-1fe368834093 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.142029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c8300-8b45-4c9b-ba8f-1fe368834093 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.142029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6c8300-8b45-4c9b-ba8f-1fe368834093 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.142759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 314/635] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:52.144207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a051415-2ff7-43e8-92e3-80c977316887 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.146253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a051415-2ff7-43e8-92e3-80c977316887 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.146253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a051415-2ff7-43e8-92e3-80c977316887 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.164316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3a051415-2ff7-43e8-92e3-80c977316887 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:52.166175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3a051415-2ff7-43e8-92e3-80c977316887 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:52.183327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.191505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd290c51-e217-4044-8c96-07531e83e4a6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.226971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d580720-230d-4899-9247-68d2d17b0976 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.227734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d580720-230d-4899-9247-68d2d17b0976 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.228028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d580720-230d-4899-9247-68d2d17b0976 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.228320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d580720-230d-4899-9247-68d2d17b0976 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.246030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 315/636] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:52.246030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a051415-2ff7-43e8-92e3-80c977316887 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.246339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d580720-230d-4899-9247-68d2d17b0976 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.246339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d580720-230d-4899-9247-68d2d17b0976 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3d580720-230d-4899-9247-68d2d17b0976 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:52.259380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:52.260561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:52.261633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:52.261633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:52.261633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:52.261815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 322/637] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:52.263059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb36612-05bd-4682-9ecf-7cbd0b0e2d0d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.263335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb36612-05bd-4682-9ecf-7cbd0b0e2d0d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.263600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb36612-05bd-4682-9ecf-7cbd0b0e2d0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.283707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2bb36612-05bd-4682-9ecf-7cbd0b0e2d0d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:52.284981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2bb36612-05bd-4682-9ecf-7cbd0b0e2d0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:52.359699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 323/638] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:52.360717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb36612-05bd-4682-9ecf-7cbd0b0e2d0d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.384744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.389958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a051415-2ff7-43e8-92e3-80c977316887 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.444605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.445237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.445504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.445768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.467894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.469348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:52.495816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:52.496820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:52.497701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:52.497701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:52.497701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:52.497701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 316/639] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:52.502978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.506745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.563991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb36612-05bd-4682-9ecf-7cbd0b0e2d0d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.605567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.606205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.606478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.606742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.606979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85431505-a568-4bab-9744-a6df0ed90871 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.607377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:52.607673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.609316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.622310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 324/640] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:52.623214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.642328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.642940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.643215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.643575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.644212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:52.644557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.646232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.659780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 317/641] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:52.664369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.714338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.729867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.735850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f7cc1a-7a5c-4375-abea-4b4e5551266d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.768078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e87fc78-8293-4793-958e-8b2b54dafb61 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.769950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.770952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.770952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.771133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.782954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:52.788573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:52.790528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 325/642] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:52.792178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.813219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.813883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.814168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.814438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.821578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.823104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:52.834362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:52.835421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 318/643] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/users/glance => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:52.836375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.903976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b124218-c305-4e00-be94-e5ff17ec588c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.933521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbff2691-6415-4ed6-9adf-8084c24c8167 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:52.940410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.941048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.941318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.941583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.948787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:52.950453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:52.953072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.957369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:52.964618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:52.965687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 326/644] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:52.971729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:52.976593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:52.977462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:52.977947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:52.978378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:52.988696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:52.994097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:52.995210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 319/645] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:52.996040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.084001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9081d803-f12b-48ba-b132-315e6b11f532 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.094999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e41fbc06-b962-4e0c-8567-04045a882a81 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.121038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.122372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.122751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.123118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.124318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:53.124630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.126066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.129143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.129945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.130396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.130827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.131744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:53.132197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.133961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.134235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 327/646] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:53.135053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.157876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 320/647] 10.222.0.61 () {56 vars in 1137 bytes} [Thu Sep 29 01:59:52 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:53.159088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.203243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.207630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.250191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8720ca4-9c13-429e-bdf8-912974fd9972 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.260350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0089721e-7af2-4211-852f-ad2b84aa11f0 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.284187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.284928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.285319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.285736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.286502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:53.286899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.288662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.295263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.295873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.296145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.296410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.297076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:53.297375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.298782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.307739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 328/648] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:53.307739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.311549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 321/649] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:53.312434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.409245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-519a78f5-c325-4148-b64f-90640d7627ec None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.412537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9605b71a-8f1a-4f45-9d56-bfa10f0b8374 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.415239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.436606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.448017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.448764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.449128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.449483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.453692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.454362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.454642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.454915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.466383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.467945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:53.482855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:53.483973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:53.485066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:53.485066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:53.485066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:53.490827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 329/650] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:53.490827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.526370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.527978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.602017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b292e4ed-eeaf-4058-a871-cf6aa614ea50 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.631544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 322/651] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:53.632883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.636510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.641173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.641785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.642089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.642360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.643362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=bc807d5f532b4e4c9953ba5e7eb878bb) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:53.643674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.645123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.651586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 330/652] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/role_assignments?user.id=bc807d5f532b4e4c9953ba5e7eb878bb&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:53.652375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.709854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.767452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a536a5d-66f3-4ea3-9ca8-0a238548ad5a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.785420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7704331-3d23-4f02-b62b-02efe7151b20 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.816523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.817740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.818512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.818977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.828382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:53.828708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:53.829124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:53.829461 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:53.830337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:53.830709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.832304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.833719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:53.835604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:53.848636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 331/653] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:53.849913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:53.853790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:53.854862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:53.856383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:53.856383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:53.856383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:53.859625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 323/654] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:53.860608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:53.923729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.974999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1910a916-88da-4bb9-ae28-364e548465db None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:53.979786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:53.983631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69131153-2d94-4b31-af06-129f36e18bc0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.018684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.019314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.019585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.020002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.027351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.028251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.028529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.028770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.028882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.030871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.041217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.042629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:54.045686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:54.046642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:54.047577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 324/655] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/users/cinder => generated 84 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:54.048171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:54.059648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:54.060656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:54.061562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:54.061562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:54.061562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:54.061562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 332/656] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:54.062596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.160871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29eafb3-4ccd-492f-83cc-f4e00822b1d3 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.175984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020b67e5-6dd9-4bbd-badc-70607c111ec4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.179074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.199279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.199886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.200162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.200425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.201060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:54.201352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.202750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.212136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c425809b-0a78-43d4-b7e5-0356f4d24b15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.212751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c425809b-0a78-43d4-b7e5-0356f4d24b15 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.213095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c425809b-0a78-43d4-b7e5-0356f4d24b15 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.213374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c425809b-0a78-43d4-b7e5-0356f4d24b15 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.214208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c425809b-0a78-43d4-b7e5-0356f4d24b15 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:54.214513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c425809b-0a78-43d4-b7e5-0356f4d24b15 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.219478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 325/657] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:54.219478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.219692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c425809b-0a78-43d4-b7e5-0356f4d24b15 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.241251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 333/658] 10.222.0.61 () {56 vars in 1137 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:54.241251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0528eee5-22eb-47ad-bd9a-9ff61437c8ce None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.241251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0528eee5-22eb-47ad-bd9a-9ff61437c8ce None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.241251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0528eee5-22eb-47ad-bd9a-9ff61437c8ce None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.244594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 334/659] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:54 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:54.245042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0528eee5-22eb-47ad-bd9a-9ff61437c8ce None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.256427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.337014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3c0fed5-557f-43d6-873a-1d24d225c893 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.357850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0528eee5-22eb-47ad-bd9a-9ff61437c8ce None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.374646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.375252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.375521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.375788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.376476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:54.376773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.378170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.394792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 326/660] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:54.394792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.399191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b287da14-cc55-4a84-85bb-1a0b19c440e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.399829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b287da14-cc55-4a84-85bb-1a0b19c440e4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.400515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b287da14-cc55-4a84-85bb-1a0b19c440e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.400795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b287da14-cc55-4a84-85bb-1a0b19c440e4 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.408915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b287da14-cc55-4a84-85bb-1a0b19c440e4 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.410728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b287da14-cc55-4a84-85bb-1a0b19c440e4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b287da14-cc55-4a84-85bb-1a0b19c440e4 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:54.425491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:54.426529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:54.427516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 335/661] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/users/neutron => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:54.428156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9ce6d-9796-408e-a499-232479b3a5fc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.428156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9ce6d-9796-408e-a499-232479b3a5fc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.428590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16a9ce6d-9796-408e-a499-232479b3a5fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.433441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.447548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-16a9ce6d-9796-408e-a499-232479b3a5fc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:54.448737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-16a9ce6d-9796-408e-a499-232479b3a5fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:54.500604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2c9b23-2feb-4b03-a1b6-b9e524cc5629 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.518613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.526356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 336/662] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:54.527765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a9ce6d-9796-408e-a499-232479b3a5fc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.547571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.548098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.548729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.548729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.634831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.637189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.648563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16a9ce6d-9796-408e-a499-232479b3a5fc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.683845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.685516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.685650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.685924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.686240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.693440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.694989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:54.706896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:54.707842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:54.708749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:54.713029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 337/663] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:54.714236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.772680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 327/664] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:54.773530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.791567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.838373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333ed58b-f66f-4908-979c-e81670df1422 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.873247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-199aa0f9-6284-4693-9e58-ddbd441e7c15 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:54.874269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.874914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.875189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.876050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.876173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:54.876434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.877853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.901300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 338/665] 10.222.0.61 () {56 vars in 1139 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:54.901936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:54.908232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:54.908840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:54.909111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:54.909376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:54.910096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:54.910396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:54.911830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:54.914787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:54.925365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 328/666] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:54.926123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.011648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd6b94-f55a-40e6-9045-138cc50c7c4f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.029625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.039183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef82435b-b8d2-4196-8844-09f4fb694442 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.049276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.049917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.050220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.050489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.061908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.064054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.075120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.075754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.076032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.076302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:55.079664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:55.080576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:55.081477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:55.081477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:55.081477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:55.081477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 339/667] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:55.082152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.091701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:55.096942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:55.098653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 329/668] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 01:59:54 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:55.100171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.174077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.202728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2629b3c9-87b6-4f30-a2df-2d656e030d46 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.223378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ba87dc5-7926-4ef6-82a6-a2c9ded07faa None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.236795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.237406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.237676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.237945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.258488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.259091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.259368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.259643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.260360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=d6cfd5241a0147d9bfe0b7addb3e1851) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:55.260661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.262124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.270159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 330/669] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/role_assignments?user.id=d6cfd5241a0147d9bfe0b7addb3e1851&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:55.270159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.279061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.337396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.339587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.415736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1531fe6-180f-4e61-a534-f146fcc28d95 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.439624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.456289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.456967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.457321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.457554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.458178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:55.458479 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.459875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.475090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 340/670] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:55.475661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 331/671] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:55.476064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.481327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.558273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.602890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca68f3ae-5f06-4a53-9f72-76d6741175aa None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.620217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7570747-56fb-40ec-a81e-d826dfd6016e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.638401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.638766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.639041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.639307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.639968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:55.640265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.641626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.650330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 341/672] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:55.651371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.663815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.665173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.666078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.666577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.675662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.677881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:55.693059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:55.694093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:55.694984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:55.701098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 332/673] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:55.701098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.750356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.780387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05562d7d-e494-4ea6-bafe-ffe59c3ba26a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.812120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e6be3c-e379-4e12-944d-5c5babe8493b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:55.817578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.826047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.826689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.826967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.827236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.841392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.843478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.850390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:55.851293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:55.851825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:55.852287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:55.859640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:55.860596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:55.862176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:55.862176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:55.862176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:55.862176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 342/674] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:55.862176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.869063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:55.871078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:55.887891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:55.888947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:55.889853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:55.889853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:55.889853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:55.892171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 333/675] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:55.893384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:55.993104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:55.997178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69791170-1d7d-4145-ade0-2694d731ea60 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.001805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa26a4d-9fe6-4f2c-9afd-2685d3565f5d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.036982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.037590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.037863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.037965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93538d73-6598-4f45-833d-b93bd89e84f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.038203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.038578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93538d73-6598-4f45-833d-b93bd89e84f1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.038776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:56.038871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93538d73-6598-4f45-833d-b93bd89e84f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.039070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.039201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93538d73-6598-4f45-833d-b93bd89e84f1 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.039997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93538d73-6598-4f45-833d-b93bd89e84f1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:56.040298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93538d73-6598-4f45-833d-b93bd89e84f1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.040528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.041714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93538d73-6598-4f45-833d-b93bd89e84f1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.052583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 334/676] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:56.053376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.065586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 343/677] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:56.065586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c94a16-0bde-475f-95c6-bda6e8e01c89 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.065586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c94a16-0bde-475f-95c6-bda6e8e01c89 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.065586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c94a16-0bde-475f-95c6-bda6e8e01c89 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.065586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 344/678] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:56.065586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c94a16-0bde-475f-95c6-bda6e8e01c89 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.112603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:56.159979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0d6cc4-55a3-4d18-9c3d-3f3c895debe5 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.182125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c94a16-0bde-475f-95c6-bda6e8e01c89 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.196102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.196719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.196992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.197257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.197889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:56.198215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.199587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.216077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 335/679] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:56.216077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.226499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-927b8859-65bf-4815-8fb4-3b33d5e021b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.227279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b8859-65bf-4815-8fb4-3b33d5e021b5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.227640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b8859-65bf-4815-8fb4-3b33d5e021b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.227812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-927b8859-65bf-4815-8fb4-3b33d5e021b5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.228296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:56.234933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927b8859-65bf-4815-8fb4-3b33d5e021b5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.236367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-927b8859-65bf-4815-8fb4-3b33d5e021b5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-927b8859-65bf-4815-8fb4-3b33d5e021b5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:56.247801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:56.248678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:56.249578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 345/680] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:56.250785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8dff4-45b3-4aa4-ae65-7a9ec7279e89 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.250785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8dff4-45b3-4aa4-ae65-7a9ec7279e89 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.250785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8be8dff4-45b3-4aa4-ae65-7a9ec7279e89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.268864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8be8dff4-45b3-4aa4-ae65-7a9ec7279e89 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:56.270033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8be8dff4-45b3-4aa4-ae65-7a9ec7279e89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:56.326499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7510da40-e9f9-4e1f-919a-2f3d2ac4b762 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.341858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 346/681] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:56.343340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8dff4-45b3-4aa4-ae65-7a9ec7279e89 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.375175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.376214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.376938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:56.377659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.378277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.379251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=36c4d127aac743baaeb86157dbbeb689) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:56.379695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.382409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.398568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 336/682] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/role_assignments?user.id=36c4d127aac743baaeb86157dbbeb689&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 608 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:56.399655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.462700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8be8dff4-45b3-4aa4-ae65-7a9ec7279e89 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.504332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.505450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.506236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.506696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.516330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.518509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:56.534625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:56.535619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 01:59:56.536548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:56.539654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 347/683] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/users/swift => generated 83 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:56.545648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:56.547787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.608203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80db25da-631f-40b1-b046-b57f950e9f4f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.652120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.652919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.653134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.653405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.654391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:56.654499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.655900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.669964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 337/684] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 271 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:56.670192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:56.670763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.678842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc90fa7-2491-455b-8a00-291ea1d1d5d5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.721933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.722612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.722886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.723151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.723883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:56.724181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.725608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.749373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 348/685] 10.222.0.61 () {56 vars in 1135 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:56.750567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.802199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d573199c-7045-4f14-91a1-24b4ada46c8d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.850182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.851252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.851875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.852140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.855588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:56.866712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 01:59:56.872542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5549a31-ade0-4388-88ad-a02123463a2e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 01:59:56.873187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:56.874159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 01:59:56.875015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:56.878077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 338/686] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:56.879092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.918516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:56.919461 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:56.919954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:56.920417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:56.921426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=cc2f041fe7194a3da199517b4cdbd265) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:56.921929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:56.923631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:56.932192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 349/687] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/role_assignments?user.id=cc2f041fe7194a3da199517b4cdbd265&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 608 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:56.933285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:56.952710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:57.053895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060de104-77f1-4ccc-b3a1-5193a5f901d7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:57.058954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ae1dd9-3381-46e8-ba97-7c7ec49ce1c8 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:57.125639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:57.128136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.128639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.129252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.130294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:57.131664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:57.135011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:57.140309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:57.142367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.143634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.146433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.154380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 350/688] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:57.161454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:57.288600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:57.315919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:57.330826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:57.332332 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:57.379036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c262d376-b3ac-46eb-86b8-1dfb766dda73 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:57.474611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:57.475251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.475519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.475780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.499268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:57.503989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:57.533284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 339/689] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:56 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 654 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:57.540968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:57.542610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:57.547150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:57.547150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:57.547150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:57.547150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 351/690] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 381 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:57.550911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:57.552163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:57.724219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf320ca-5544-49eb-a793-b5f06124530d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:57.768872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:57.777208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd8680cd-d8b4-4b4a-a661-3119f2fa7dd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:57.777936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8680cd-d8b4-4b4a-a661-3119f2fa7dd9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.778331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8680cd-d8b4-4b4a-a661-3119f2fa7dd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.778713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd8680cd-d8b4-4b4a-a661-3119f2fa7dd9 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.779472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8680cd-d8b4-4b4a-a661-3119f2fa7dd9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:57.779855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8680cd-d8b4-4b4a-a661-3119f2fa7dd9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:57.781315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd8680cd-d8b4-4b4a-a661-3119f2fa7dd9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:57.789293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:57.797675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 352/691] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:57.797675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2edd5-e26f-4a77-8988-7c0c3b0a9406 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.797675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2edd5-e26f-4a77-8988-7c0c3b0a9406 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.797675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7c2edd5-e26f-4a77-8988-7c0c3b0a9406 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.798256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ee88ecf-1c4d-464d-a267-ac8cdee1d8f5 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:57.798335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 353/692] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:57.799312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c2edd5-e26f-4a77-8988-7c0c3b0a9406 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:57.839419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8ad2c73-f07a-401d-aa70-9b674b1d612d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:57.839801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ad2c73-f07a-401d-aa70-9b674b1d612d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.840078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ad2c73-f07a-401d-aa70-9b674b1d612d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.840341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ad2c73-f07a-401d-aa70-9b674b1d612d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.851092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ad2c73-f07a-401d-aa70-9b674b1d612d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:57.852991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8ad2c73-f07a-401d-aa70-9b674b1d612d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a8ad2c73-f07a-401d-aa70-9b674b1d612d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:57.868995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:57.870605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:57.872526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:57.872526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:57.872526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:57.875027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 340/693] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 01:59:57.876514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7333a69-e5dd-4661-a42f-d02caec8dde1 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.876787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7333a69-e5dd-4661-a42f-d02caec8dde1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.877059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7333a69-e5dd-4661-a42f-d02caec8dde1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.896384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b7333a69-e5dd-4661-a42f-d02caec8dde1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:57.897566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b7333a69-e5dd-4661-a42f-d02caec8dde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:57.927338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7c2edd5-e26f-4a77-8988-7c0c3b0a9406 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:57.964345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:57.965387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:57.965858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:57.966385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:57.974085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:57.974414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 341/694] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:57.975409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7333a69-e5dd-4661-a42f-d02caec8dde1 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:57.976286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 01:59:57.989913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:57.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 01:59:57.992006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:57.994692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 354/695] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 01:59:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:58.002407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.104529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.141501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.163188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-653990d0-75d1-44bc-a08c-8482f2557f7b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.199821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.200776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.201246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.201731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.202705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:58.203288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.204928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.216077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7333a69-e5dd-4661-a42f-d02caec8dde1 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.223635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 355/696] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:58.235025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.310750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.310993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.311261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.311501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.312152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:58.312448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.314022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.326601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 342/697] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 351 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:58.327509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.339173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-027df166-f2a9-4a1f-8cd2-0f56d07e5291 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.347688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.378673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.379199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.379733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.379733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.453426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e85da30-a882-4d30-8819-063f2ff18ded None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.473496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.475869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.489376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.489787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.490083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.490350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.498467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.502430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.504104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:58.519946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:58.520963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:58.521885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:58.521885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:58.521885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:58.526092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 343/698] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:58.527020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.623199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 356/699] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 29 01:59:58.624905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.635669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.640962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6650d89f-f3dc-4510-a8c3-4c0e19d8891e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.676981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.677689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.677964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.678321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.686152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.687600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:58.699828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:58.700729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 344/700] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:58.702224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.753507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.763419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672f0cd7-b156-4316-a92c-b49cc59b6d85 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.806484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c54f46dc-3efc-42f9-88cb-c384e0f0449f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:58.811735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.812618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.813089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.813553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.814925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:58.815467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.817038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.834751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 357/701] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:58.836220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.848202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:58.848838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:58.849111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:58.849380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:58.861328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:58.862845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 01:59:58.877359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:58.878410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 01:59:58.879300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 01:59:58.879300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 01:59:58.879300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:58.883505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 345/702] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:58.884496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:58.934643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:58.981024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eddc87b-b0fd-42d4-97f6-bcbb7181720b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.003693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06e2f9bd-b22e-4dc3-a57a-2e6ecb990d75 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.006110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:59.017574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.018375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.018655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.018923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.019612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:59.019908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.021345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.038055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 358/703] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:59.038055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:59.042128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c9ec7c7-f946-4da8-9c65-878a3fee7dbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.042750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ec7c7-f946-4da8-9c65-878a3fee7dbe None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.043015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ec7c7-f946-4da8-9c65-878a3fee7dbe None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.043282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c9ec7c7-f946-4da8-9c65-878a3fee7dbe None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.043920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9ec7c7-f946-4da8-9c65-878a3fee7dbe None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.045352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c9ec7c7-f946-4da8-9c65-878a3fee7dbe None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.069475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 346/704] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 01:59:58 2022] POST /identity/v3/users => generated 254 bytes in 185 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 01:59:59.071277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309679a7-2fa9-4d59-9925-36ae1ec4d2ec None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.071773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309679a7-2fa9-4d59-9925-36ae1ec4d2ec None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.072241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-309679a7-2fa9-4d59-9925-36ae1ec4d2ec None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.074093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 347/705] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:59 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:59.075177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309679a7-2fa9-4d59-9925-36ae1ec4d2ec None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:59.165095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e730c6b3-c80b-4ba5-ac42-71370def43bc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.193550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-309679a7-2fa9-4d59-9925-36ae1ec4d2ec None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.196323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:59.213553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0c1ddf-bbfd-439a-84cd-06389c539727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.214506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0c1ddf-bbfd-439a-84cd-06389c539727 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.214506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0c1ddf-bbfd-439a-84cd-06389c539727 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.214776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0c1ddf-bbfd-439a-84cd-06389c539727 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.215396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0c1ddf-bbfd-439a-84cd-06389c539727 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:59.215696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0c1ddf-bbfd-439a-84cd-06389c539727 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.217219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0c1ddf-bbfd-439a-84cd-06389c539727 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.235419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 359/706] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:59.235419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82869f6e-816d-479d-b0bc-6b8e5e4ba8b5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.235419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82869f6e-816d-479d-b0bc-6b8e5e4ba8b5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.235419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82869f6e-816d-479d-b0bc-6b8e5e4ba8b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.251250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.252561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.253048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.253542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.254707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=34b928e9fe314d0fb61122b7a558c1b7) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:59.255317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.257205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.258816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-82869f6e-816d-479d-b0bc-6b8e5e4ba8b5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:59.260514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-82869f6e-816d-479d-b0bc-6b8e5e4ba8b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:59.266416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 348/707] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/role_assignments?user.id=34b928e9fe314d0fb61122b7a558c1b7&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 608 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:59.267618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:59.321960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:59.373933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 360/708] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 01:59:59.379652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35369faf-724f-4270-aa0c-b28d1ed4d334 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.380283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35369faf-724f-4270-aa0c-b28d1ed4d334 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.380283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35369faf-724f-4270-aa0c-b28d1ed4d334 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.385354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 361/709] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 01:59:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 01:59:59.389545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2f6bb-c16e-48ee-90ba-e3760e262e5d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.389829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2f6bb-c16e-48ee-90ba-e3760e262e5d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.390142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c2f6bb-c16e-48ee-90ba-e3760e262e5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.413415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f5c2f6bb-c16e-48ee-90ba-e3760e262e5d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 01:59:59.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f5c2f6bb-c16e-48ee-90ba-e3760e262e5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 01:59:59.421589 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c82ea6a-bfd0-4574-8ea1-e2436ac6205e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.466320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.466919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.467195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.467467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.476036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.477818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 01:59:59.492302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:59.493336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 01:59:59.494323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 349/710] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:59.502082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:59.516495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 362/711] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 01:59:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 01:59:59.608845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:59.629440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1958c446-46f6-4400-928d-32fba34da024 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.664258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.665012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.665286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.665598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.666319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:59.666622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.668076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.681263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 350/712] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 01:59:59.686879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:59.687010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c2f6bb-c16e-48ee-90ba-e3760e262e5d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:59.759106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:59.811854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66088788-b18e-4869-bda2-10dae703e9b9 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.832660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c2f6bb-c16e-48ee-90ba-e3760e262e5d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 01:59:59.846718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.847493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.847772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.848415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.849425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=bc807d5f532b4e4c9953ba5e7eb878bb) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 01:59:59.849738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.851721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.859129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 351/713] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/role_assignments?user.id=bc807d5f532b4e4c9953ba5e7eb878bb&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 01:59:59.860065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 01:59:59.869689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 01:59:59.870377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 01:59:59.870649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 01:59:59.870917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 01:59:59.871508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 01:59:59.872910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 01:59:59.874355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 01:59:59.886049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 01:59:59.887274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 01:59:59.888145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 363/714] 10.222.0.61 () {56 vars in 1129 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 01:59:59.892830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:00.018750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b604afa0-ed45-4937-b534-f43331444cf7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:00.063812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa5d09c-4813-4566-8e1b-58a740b556bd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:00.068446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:00.069340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.069808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.070293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] PATH_INFO: `/v3/services/network` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.071034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:00.073044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:00.086816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:00.088208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:00.089175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:00.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:00.092644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 352/715] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/services/network => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:00.103467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:00.133288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:00.134547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.134971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.135470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.136358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:00.136843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:00.138517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:00.144387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 364/716] 10.222.0.61 () {56 vars in 1124 bytes} [Thu Sep 29 01:59:59 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:00.149728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:00.227047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:00.264585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e06349b-28ae-4d5c-ad7c-5c2523b7d948 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:00.288457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42af831-c4e7-4b80-ae92-686377f86457 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:00.301279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:00.302201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.302655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.303095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.303928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:00.304460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:00.306481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:00.313111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 365/717] 10.222.0.61 () {56 vars in 1124 bytes} [Thu Sep 29 02:00:00 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:00.337319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1e270bf-17da-4e29-bbe6-f3bc944460a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:00.337953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e270bf-17da-4e29-bbe6-f3bc944460a4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.338251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e270bf-17da-4e29-bbe6-f3bc944460a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.338516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1e270bf-17da-4e29-bbe6-f3bc944460a4 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.339163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e270bf-17da-4e29-bbe6-f3bc944460a4 None admin] RBAC: Adding query filter params (name=network) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:00.339461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e270bf-17da-4e29-bbe6-f3bc944460a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:00.340917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1e270bf-17da-4e29-bbe6-f3bc944460a4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:00.346678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 353/718] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:00 2022] GET /identity/v3/services?name=network => generated 124 bytes in 244 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:00.352558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:00.423178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:00.491508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96aeff78-0d54-4418-9e29-75ce6bcdf1f0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:00.529800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b101281a-d85d-484f-a619-a82767485e89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:00.530851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b101281a-d85d-484f-a619-a82767485e89 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:00.531320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b101281a-d85d-484f-a619-a82767485e89 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:00.531784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b101281a-d85d-484f-a619-a82767485e89 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:00.532606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b101281a-d85d-484f-a619-a82767485e89 None admin] RBAC: Adding query filter params (type=network) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:00.533112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b101281a-d85d-484f-a619-a82767485e89 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:00.534810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b101281a-d85d-484f-a619-a82767485e89 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:00.542295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 366/719] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:00 2022] GET /identity/v3/services?type=network => generated 124 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:02.056436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d940d6-00b9-42db-9941-5a1651e2106e None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.057002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d940d6-00b9-42db-9941-5a1651e2106e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.057002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d940d6-00b9-42db-9941-5a1651e2106e None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.058593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 354/720] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:02.063948 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6722475-9a4b-459a-902b-91e9be9fe063 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.064353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6722475-9a4b-459a-902b-91e9be9fe063 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.064538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6722475-9a4b-459a-902b-91e9be9fe063 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.083221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c6722475-9a4b-459a-902b-91e9be9fe063 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:02.084412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c6722475-9a4b-459a-902b-91e9be9fe063 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:02.167955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 367/721] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 29 02:00:02.389602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2cf727-685c-4e9b-92c1-5ec1653bab40 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.390331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2cf727-685c-4e9b-92c1-5ec1653bab40 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.390816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b2cf727-685c-4e9b-92c1-5ec1653bab40 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.392575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 355/722] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:02.398394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeab525-9503-4bdb-a220-cd7660eaf077 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.398677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeab525-9503-4bdb-a220-cd7660eaf077 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.399042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daeab525-9503-4bdb-a220-cd7660eaf077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.419046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-daeab525-9503-4bdb-a220-cd7660eaf077 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:02.420250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-daeab525-9503-4bdb-a220-cd7660eaf077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:02.453370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:02.505262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 368/723] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 02:00:02.506562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2cf727-685c-4e9b-92c1-5ec1653bab40 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:02.671352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b2cf727-685c-4e9b-92c1-5ec1653bab40 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:02.717664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e23e9940-cb51-474a-97e5-3e9f7c178f60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:02.718350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e9940-cb51-474a-97e5-3e9f7c178f60 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.718624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e9940-cb51-474a-97e5-3e9f7c178f60 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.718892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e23e9940-cb51-474a-97e5-3e9f7c178f60 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.721018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:02.734994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e23e9940-cb51-474a-97e5-3e9f7c178f60 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-e23e9940-cb51-474a-97e5-3e9f7c178f60 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:02.743072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:02.743998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:02.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 356/724] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 02:00:02 2022] GET /identity/v3/roles/service => generated 85 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:02.751118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeab525-9503-4bdb-a220-cd7660eaf077 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:02.806063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf3f57-238d-4fc7-91ad-0666e8308d37 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.806395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf3f57-238d-4fc7-91ad-0666e8308d37 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.806654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbf3f57-238d-4fc7-91ad-0666e8308d37 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.808821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 357/725] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:02 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:02.813743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aec636-8025-465f-9f69-9bb948a6fb7e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.814032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aec636-8025-465f-9f69-9bb948a6fb7e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.814301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7aec636-8025-465f-9f69-9bb948a6fb7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.835019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a7aec636-8025-465f-9f69-9bb948a6fb7e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:02.836229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a7aec636-8025-465f-9f69-9bb948a6fb7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:02.909829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daeab525-9503-4bdb-a220-cd7660eaf077 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:02.918100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 358/726] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:02 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 02:00:02.919012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aec636-8025-465f-9f69-9bb948a6fb7e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:02.971326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:02.971854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:02.974653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:02.974928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:02.975608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:02.977973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:02.979398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.001643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 369/727] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 02:00:02 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:03.007059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.061961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.086468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7aec636-8025-465f-9f69-9bb948a6fb7e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.118442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.123367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63e77b2b-b096-452b-acbc-ab769864f895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.124545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e77b2b-b096-452b-acbc-ab769864f895 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.125011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e77b2b-b096-452b-acbc-ab769864f895 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.125475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63e77b2b-b096-452b-acbc-ab769864f895 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.137248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63e77b2b-b096-452b-acbc-ab769864f895 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-63e77b2b-b096-452b-acbc-ab769864f895 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:03.143197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:03.144121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:03.144982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:03.148699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 359/728] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 230 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:03.151050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362272cd-0a71-4095-ac11-ca03b9170587 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.151539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362272cd-0a71-4095-ac11-ca03b9170587 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.152016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362272cd-0a71-4095-ac11-ca03b9170587 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.154483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 360/729] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:03 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:03.155388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362272cd-0a71-4095-ac11-ca03b9170587 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.194618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f84f637f-5fcb-4ce8-9d89-303840d11806 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.235697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5f892d-b349-4580-adc3-ca280f1b0653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.236438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f892d-b349-4580-adc3-ca280f1b0653 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.236765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f892d-b349-4580-adc3-ca280f1b0653 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.237525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5f892d-b349-4580-adc3-ca280f1b0653 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.248686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5f892d-b349-4580-adc3-ca280f1b0653 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:03.250178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df5f892d-b349-4580-adc3-ca280f1b0653 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-df5f892d-b349-4580-adc3-ca280f1b0653 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:03.265626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:03.266669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:03.267556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:03.267556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:03.267556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:03.267556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 370/730] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:03.268374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76908c0e-d1e2-4483-829a-dc437f630669 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.268656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76908c0e-d1e2-4483-829a-dc437f630669 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.268929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76908c0e-d1e2-4483-829a-dc437f630669 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.278806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362272cd-0a71-4095-ac11-ca03b9170587 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.287625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-76908c0e-d1e2-4483-829a-dc437f630669 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:03.288857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-76908c0e-d1e2-4483-829a-dc437f630669 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:03.314109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.314722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.315068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.315333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.315986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:03.316276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:03.317641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.325825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 361/731] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:03.326643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.374145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 371/732] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 02:00:03.375029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76908c0e-d1e2-4483-829a-dc437f630669 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.384928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.385438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.447456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a35b80-c125-4815-b3fb-8d2ff4b41e3e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.486318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.487359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.487878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.488457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.494741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76908c0e-d1e2-4483-829a-dc437f630669 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.500788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:03.507315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:03.508251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:03.509068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:03.511872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 362/733] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:03.513019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.546553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.547174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.547451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.550021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.550021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:03.550021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:03.550772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.563206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 372/734] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:03.564153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.638454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.639154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4528aeed-5429-4c19-b41e-f06b7678d1f3 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.647003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.674917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82a15622-26b7-435a-96d0-33c01daf0a6e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.676634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.677263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.677533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.677797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.688251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:03.689693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:03.703649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:03.704595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:03.705901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:03.705901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:03.705901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:03.705901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 363/735] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:03.705901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.713352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-822cdb30-ee21-418e-8c89-505b4aebcd23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.713964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822cdb30-ee21-418e-8c89-505b4aebcd23 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.714269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822cdb30-ee21-418e-8c89-505b4aebcd23 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.714540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-822cdb30-ee21-418e-8c89-505b4aebcd23 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.715204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822cdb30-ee21-418e-8c89-505b4aebcd23 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:03.715503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822cdb30-ee21-418e-8c89-505b4aebcd23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:03.716860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-822cdb30-ee21-418e-8c89-505b4aebcd23 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.724999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 373/736] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:03.726242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b638c-f11c-44fb-b204-26c3081aa408 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.726526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b638c-f11c-44fb-b204-26c3081aa408 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.726794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b638c-f11c-44fb-b204-26c3081aa408 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.727970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 374/737] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:03.728641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b638c-f11c-44fb-b204-26c3081aa408 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.807183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c6dcbeb-705b-43db-b8cc-2893a4290ae8 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.829152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b638c-f11c-44fb-b204-26c3081aa408 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:03.840356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.840986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.841257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.841524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.848964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:03.850487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.855134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:03.861853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:03.862893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:03.863817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:03.864349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:03.864436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 364/738] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:03.864504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:03.864571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.864636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:03.864636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:03.864765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:03.866134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:03.886389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 375/739] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:03 2022] POST /identity/v3/services => generated 251 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:03.887211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:03.891984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:03.988239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4908492-c97c-426e-887c-9f5bc94cbc7d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.018217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cb2ac90-d262-418d-b440-f52370c4f15b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.025056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a3c3c4-597b-4438-82f2-0c51349facbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.025681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a3c3c4-597b-4438-82f2-0c51349facbf None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.025954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a3c3c4-597b-4438-82f2-0c51349facbf None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.026249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a3c3c4-597b-4438-82f2-0c51349facbf None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.026891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a3c3c4-597b-4438-82f2-0c51349facbf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:04.027307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a3c3c4-597b-4438-82f2-0c51349facbf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.028729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a3c3c4-597b-4438-82f2-0c51349facbf None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.046691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 365/740] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:04.046691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a898483-dfd5-4a23-931e-aac14abd8db7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.046691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a898483-dfd5-4a23-931e-aac14abd8db7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.046691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a898483-dfd5-4a23-931e-aac14abd8db7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.064573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.065476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.065476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.065755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.066271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6a898483-dfd5-4a23-931e-aac14abd8db7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:04.067491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6a898483-dfd5-4a23-931e-aac14abd8db7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:04.077334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.078190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:04.094078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:04.095252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:04.096128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:04.096128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:04.096128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:04.096663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 376/741] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:04.097643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.137014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.152075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 366/742] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:04 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 29 02:00:04.152075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a898483-dfd5-4a23-931e-aac14abd8db7 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.165111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.217176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ea09136-750f-422b-85b0-44beb38deded None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.253351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.253968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.254395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.254673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.255406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:04.255724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.257257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.258215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a898483-dfd5-4a23-931e-aac14abd8db7 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.280665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 377/743] 10.222.0.61 () {56 vars in 1149 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:04.281595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.300652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.302233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.302684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.303117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] PATH_INFO: `/v3/users/swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.311821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.313362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:04.325433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:04.326610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:04.327528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 367/744] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/users/swift => generated 83 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:04.328099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.391990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.400720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cfafc2e-88d9-47cb-8ee0-9aa8765f7756 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.411114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.441982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.443232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.444050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.444739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.445697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:04.446275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.447956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.453571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb8d0ea4-fbe8-4e03-9064-2571deed11bb None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.466277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 378/745] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:04.467398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.488909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.489518 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.489792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.490094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.562147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.563625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.579021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b34478d-7f74-40aa-a24b-7c35bc4df5b7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.616261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.617200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.617561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.617855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.618602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:04.618900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.620337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.632710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.642479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 379/746] 10.222.0.61 () {56 vars in 1135 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:04.642479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.659529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.670288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 368/747] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 29 02:00:04.670962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.753286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c798557-00e3-426c-8c55-204aad1bb39d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.775339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebc8f50-34a0-43ec-8c5d-02df530be82d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.787158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.787753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.788036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.788318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.788989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.790415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.809572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 380/748] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:04 2022] POST /identity/v3/services => generated 240 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:04.810448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.825763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:04.827095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:04.827622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:04.828087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:04.837401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:04.839427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:04.855267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:04.856319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:04.857307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:04.859850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 369/749] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/users/nova => generated 82 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:04.860842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:04.900209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.914032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:04.952206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b097565f-1aff-4fdf-9766-45f583022b53 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:04.999915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.000689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.001081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.001450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.017149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf33c171-63de-44bf-ad4d-a27009e67c95 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.057098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.058032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.058306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.058572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.070821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.072085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.083906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.085439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:05.087795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:05.088707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:05.089783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:05.089783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:05.089783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:05.090022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 370/750] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:05.095207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.172754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:05.214036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 381/751] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:04 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 399 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:05.214236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b771650e-0e0e-47bf-bc6d-1c46a16d9e36 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.214686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:05.215344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.255993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.256609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.256882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.257344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.258275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:05.258584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.260041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.284186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 371/752] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:05.286405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.331236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2dc3828-b29d-4fd8-a129-3d1de88e8e18 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.368722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.369651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.370179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.370668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.371494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:05.371985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.373648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.385973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 382/753] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:05.387176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.412614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102b993-039c-40ff-a3e0-cae75cdd03aa None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.423193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:05.451124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.452134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.452601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.453456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.456328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:05.468823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.470735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:05.488926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:05.489915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:05.490842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:05.490842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:05.490842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:05.490842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 372/754] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:05.490842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.522803 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0181e1-b482-4382-b43c-4b490e127b13 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.570943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.571587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.572099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.572383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.586964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.591487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:05.611595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:05.612797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:05.613885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:05.613885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:05.613885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:05.621510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 383/755] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:05.622511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.628176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7006a26-aef5-40be-9d06-fa377f14f3be None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.670684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.672283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.672656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.673019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.681646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.683859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:05.698268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:05.699402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:05.700484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:05.702791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 373/756] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:05.703942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.733374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:05.751243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ffd7aac-60b2-4a7f-949d-8e6e85875f2c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.760348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:05.789115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.789964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.790427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.790843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.791619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:05.792055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.793658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.813555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 384/757] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:05.814925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.823167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429140cf-8d07-4759-a47c-b8315ab97184 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.859482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.860291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.860739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.861066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.861800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:05.862241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.863820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.881539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 374/758] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:05.882933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.922255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62a08740-9f71-4542-8428-9f4e4a0d7978 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:05.964165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:05.965287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:05.965682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:05.966082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:05.967232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:05.967652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:05.969828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:05.988544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 385/759] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:05.989937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:05.995445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:05.995898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-430129d3-a49e-4b58-a30d-9861834b24f7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:06.007441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:06.042645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:06.043603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.044015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.044424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.053352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:06.056225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:06.070924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:06.072041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:06.073154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:06.080440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 375/760] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:06.081994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:06.121554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2095b17-b2a2-470d-82eb-f5771cd67b17 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:06.157393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:06.157962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.158265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.158535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.165824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:06.167373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:06.179881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:06.180918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:06.181946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 386/761] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:05 2022] GET /identity/v3/projects/service => generated 88 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:06.182493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:06.200793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4180f2-6bca-4295-8b6c-22c7ee434a8b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:06.235200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b481f9-1271-4c93-bf1a-673a5909aa0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:06.235824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b481f9-1271-4c93-bf1a-673a5909aa0d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.236070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b481f9-1271-4c93-bf1a-673a5909aa0d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.236339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b481f9-1271-4c93-bf1a-673a5909aa0d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.237069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b481f9-1271-4c93-bf1a-673a5909aa0d None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=d505d9008c9540659a3de626874c2457) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:06.237374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b481f9-1271-4c93-bf1a-673a5909aa0d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:06.238836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b481f9-1271-4c93-bf1a-673a5909aa0d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:06.244588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 376/762] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:06 2022] GET /identity/v3/role_assignments?user.id=d505d9008c9540659a3de626874c2457&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:06.245800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39467871-a76a-40b2-824b-9d5cb37dd2b7 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.246116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39467871-a76a-40b2-824b-9d5cb37dd2b7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.246388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39467871-a76a-40b2-824b-9d5cb37dd2b7 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.250121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 377/763] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:06 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:06.250121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39467871-a76a-40b2-824b-9d5cb37dd2b7 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:06.255744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:06.286519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:06.309602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-981324c2-5d97-4727-92a9-ae8d0c9fa6af None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:06.358898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07ac06e8-b11d-4e1e-8bc9-fb1558cc298d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:06.359680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac06e8-b11d-4e1e-8bc9-fb1558cc298d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.360047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac06e8-b11d-4e1e-8bc9-fb1558cc298d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.360418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07ac06e8-b11d-4e1e-8bc9-fb1558cc298d None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.361230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac06e8-b11d-4e1e-8bc9-fb1558cc298d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:06.361654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac06e8-b11d-4e1e-8bc9-fb1558cc298d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:06.363924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07ac06e8-b11d-4e1e-8bc9-fb1558cc298d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:06.378596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39467871-a76a-40b2-824b-9d5cb37dd2b7 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:06.382111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 387/764] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:06.384075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d971211-959a-480a-82b8-bbd29d24ef24 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.386566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d971211-959a-480a-82b8-bbd29d24ef24 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.386953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d971211-959a-480a-82b8-bbd29d24ef24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.412335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7d971211-959a-480a-82b8-bbd29d24ef24 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:06.416676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7d971211-959a-480a-82b8-bbd29d24ef24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:06.424580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:06.425488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.425904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.426368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.427317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:06.427766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:06.430064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:06.443513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 378/765] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:06.444728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:06.511979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 388/766] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:06 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:06.513186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d971211-959a-480a-82b8-bbd29d24ef24 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:06.599384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:06.617259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae251ed3-125e-40ec-8660-f59dc2d58adb None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:06.621973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:06.681496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:06.682038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.682317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.682585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.683312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=34b928e9fe314d0fb61122b7a558c1b7) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:06.683615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:06.685062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:06.701214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 379/767] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:06 2022] GET /identity/v3/role_assignments?user.id=34b928e9fe314d0fb61122b7a558c1b7&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:06.707604 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d971211-959a-480a-82b8-bbd29d24ef24 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:06.752817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:06.753825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:06.754268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:06.754677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/bc807d5f532b4e4c9953ba5e7eb878bb/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:06.759630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=bc807d5f532b4e4c9953ba5e7eb878bb, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:06.761106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:06.794793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 389/768] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:06 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/bc807d5f532b4e4c9953ba5e7eb878bb/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:00:06.852870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:06.976765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-410c2318-6c6c-46e5-aece-0ec73addd590 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:07.014911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286bff62-324f-43fd-9cef-9022e16ea3b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:07.016062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286bff62-324f-43fd-9cef-9022e16ea3b4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.016565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286bff62-324f-43fd-9cef-9022e16ea3b4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.017067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286bff62-324f-43fd-9cef-9022e16ea3b4 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.017904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286bff62-324f-43fd-9cef-9022e16ea3b4 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:07.019616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286bff62-324f-43fd-9cef-9022e16ea3b4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-286bff62-324f-43fd-9cef-9022e16ea3b4 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:07.036017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:07.037103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 02:00:07.038021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:07.042424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 380/769] 10.222.0.61 () {56 vars in 1129 bytes} [Thu Sep 29 02:00:06 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:07.049043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:07.157967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b211ce-5945-45c4-a2dc-0b1a0a97733b None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.158230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b211ce-5945-45c4-a2dc-0b1a0a97733b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.158512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b211ce-5945-45c4-a2dc-0b1a0a97733b None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.160144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 381/770] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:07.164938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4fbf243-5439-40fc-814d-10fd54a65d76 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:07.167436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6135dd76-74e2-401c-8047-a6003b90bf4e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.167765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6135dd76-74e2-401c-8047-a6003b90bf4e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.168103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6135dd76-74e2-401c-8047-a6003b90bf4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.181461 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:07.189009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6135dd76-74e2-401c-8047-a6003b90bf4e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:07.190357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6135dd76-74e2-401c-8047-a6003b90bf4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:07.204711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:07.205330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.205613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.205888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.207731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:07.208349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:07.208652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:07.210218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:07.216764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 390/771] 10.222.0.61 () {56 vars in 1124 bytes} [Thu Sep 29 02:00:07 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:07.226045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:07.313262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 382/772] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:07 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:07.386728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b7e5923-2f2f-4979-9560-2ea11fc44e79 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:07.436685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:07.437556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.438026 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.438442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.439218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:07.439660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:07.441180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:07.448462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 391/773] 10.222.0.61 () {56 vars in 1124 bytes} [Thu Sep 29 02:00:07 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:07.454401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6135dd76-74e2-401c-8047-a6003b90bf4e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:07.592121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6135dd76-74e2-401c-8047-a6003b90bf4e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:07.639587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:07.640206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.640474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.640735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.641494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] RBAC: Adding query filter params (interface=public, service_id=6e0d64703e2840cb86714b990fe2ac48, region_id=RegionOne) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:07.641799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:07.643268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:07.652397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 383/774] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:00:07 2022] GET /identity/v3/endpoints?service_id=6e0d64703e2840cb86714b990fe2ac48&interface=public®ion_id=RegionOne => generated 194 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:07.658144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:07.706953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:07.765670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ed1f5a-014b-4afb-91f7-60361de5fbce None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:07.801014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:07.802082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:07.802562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:07.803055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:07.803969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:07.806037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:07.812777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 392/775] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:07 2022] GET /identity/v3/services => generated 772 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:07.838371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:07.856994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:07.971040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea901c9-dea8-4376-a866-6ca543440d2b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:08.022070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ea6870-736e-488b-b139-76af62781dca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:08.023263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea6870-736e-488b-b139-76af62781dca None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:08.023761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea6870-736e-488b-b139-76af62781dca None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:08.024254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00ea6870-736e-488b-b139-76af62781dca None admin] PATH_INFO: `/v3/services/network` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:08.025053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ea6870-736e-488b-b139-76af62781dca None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:08.026763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00ea6870-736e-488b-b139-76af62781dca None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-00ea6870-736e-488b-b139-76af62781dca None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:08.041031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:08.042111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:08.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:08.045529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 384/776] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:07 2022] GET /identity/v3/services/network => generated 88 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:08.051606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:08.164511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42f1c4f0-9fcc-4b0f-b1de-dc90d668448b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:08.173120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:08.211314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:08.211928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:08.212208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:08.212474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:08.213105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] RBAC: Adding query filter params (name=network) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:08.213406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:08.214807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:08.220014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 393/777] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:08 2022] GET /identity/v3/services?name=network => generated 124 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:08.228646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ea6870-736e-488b-b139-76af62781dca None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:08.349347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:08.376113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00ea6870-736e-488b-b139-76af62781dca None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:08.423907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:08.424435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:08.424710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:08.424976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:08.425599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] RBAC: Adding query filter params (type=network) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:08.425956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:08.427363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:08.437619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 385/778] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:08 2022] GET /identity/v3/services?type=network => generated 350 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:08.443500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:08.608690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a884bbb-8bfb-4f1f-ab70-d4ff638e1869 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:08.658734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28691cee-0d95-458c-9118-36c5f32508f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:08.659341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28691cee-0d95-458c-9118-36c5f32508f3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:08.659611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28691cee-0d95-458c-9118-36c5f32508f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:08.659877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28691cee-0d95-458c-9118-36c5f32508f3 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:08.660569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28691cee-0d95-458c-9118-36c5f32508f3 None admin] RBAC: Adding query filter params (interface=public, service_id=f4159ba0e0c04236827f86961446d3b3, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:08.660875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28691cee-0d95-458c-9118-36c5f32508f3 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:08.662284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28691cee-0d95-458c-9118-36c5f32508f3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:08.676982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 394/779] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:00:08 2022] GET /identity/v3/endpoints?service_id=f4159ba0e0c04236827f86961446d3b3&interface=public®ion_id=RegionOne => generated 194 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:08.685746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:08.835413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57caeee-314b-49ed-a41d-9b471b871af9 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:08.886751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9646dd-31d8-44df-846c-3b47293b9d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:08.887392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9646dd-31d8-44df-846c-3b47293b9d3c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:08.887667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9646dd-31d8-44df-846c-3b47293b9d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:08.887933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9646dd-31d8-44df-846c-3b47293b9d3c None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:08.888549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9646dd-31d8-44df-846c-3b47293b9d3c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:08.889952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9646dd-31d8-44df-846c-3b47293b9d3c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:08.896018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 386/780] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:08 2022] GET /identity/v3/services => generated 772 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:09.200607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690a678-dccb-4a58-aaa0-58d3bdd7eacf None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.201318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690a678-dccb-4a58-aaa0-58d3bdd7eacf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.201812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3690a678-dccb-4a58-aaa0-58d3bdd7eacf None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.204166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 395/781] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:09.209472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0104f1ae-6649-4fd5-844b-0886f7e1a516 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.210276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0104f1ae-6649-4fd5-844b-0886f7e1a516 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.210742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0104f1ae-6649-4fd5-844b-0886f7e1a516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.223405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:09.237168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0104f1ae-6649-4fd5-844b-0886f7e1a516 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:09.238719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0104f1ae-6649-4fd5-844b-0886f7e1a516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:09.327261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 387/782] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:09.330025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701885fc-8cbe-4f4e-869f-5dd0977242b9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.330025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701885fc-8cbe-4f4e-869f-5dd0977242b9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.330025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701885fc-8cbe-4f4e-869f-5dd0977242b9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.334183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 396/783] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:09.337588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b5777-f35e-4154-9c6c-607b67c90026 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.338157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b5777-f35e-4154-9c6c-607b67c90026 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.338622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa8b5777-f35e-4154-9c6c-607b67c90026 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.368736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-aa8b5777-f35e-4154-9c6c-607b67c90026 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:09.370702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-aa8b5777-f35e-4154-9c6c-607b67c90026 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:09.466819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 388/784] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 130 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:09.491094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6e733-4138-4dec-9cbb-bf54201c5de2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.491756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6e733-4138-4dec-9cbb-bf54201c5de2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.491756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20e6e733-4138-4dec-9cbb-bf54201c5de2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.493300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 397/785] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:09.498505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c674c70-2426-4b5b-9df5-0efe1a0049bd None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.498793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c674c70-2426-4b5b-9df5-0efe1a0049bd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.499066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c674c70-2426-4b5b-9df5-0efe1a0049bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.524730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9c674c70-2426-4b5b-9df5-0efe1a0049bd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:09.525913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9c674c70-2426-4b5b-9df5-0efe1a0049bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:09.616052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e6e733-4138-4dec-9cbb-bf54201c5de2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:09.621181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:09.638863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 389/786] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:09 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:09.678346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:09.768436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20e6e733-4138-4dec-9cbb-bf54201c5de2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:09.816500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:09.817414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:09.817825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:09.818321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:09.838850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:09.841523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c674c70-2426-4b5b-9df5-0efe1a0049bd None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:09.844017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:09.845106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:00:09.846183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:09.848362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 398/787] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 02:00:09 2022] GET /identity/v3/roles/service => generated 85 bytes in 233 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:09.854235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.015055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c674c70-2426-4b5b-9df5-0efe1a0049bd None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.025708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.035511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf8b60f-8216-4ea4-bcbf-4e62e50ff62a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.067599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.068481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.068893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.069381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.078592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.079335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.079699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.080082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.080901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:10.081295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.081562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:10.082900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:10.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:10.088167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:10.088992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:10.091375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 390/788] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:09 2022] GET /identity/v3/roles/admin => generated 83 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:10.092282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.095650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 399/789] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 02:00:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:10.100164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.175764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.226774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd4b760-5baf-417c-b06f-5719dc0dcafc None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.240235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2186d6c4-d671-47e6-8571-aa5d96ddf12c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.262676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.263320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.263592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.263859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.274749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:10.279562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:10.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:10.281359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:10.285633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 391/790] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:10.286594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.289055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.289917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.290423 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.290889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.291884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:10.292390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.294086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.305595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 400/791] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:10.306915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.331907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.428792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21690835-2277-4d0f-9f78-8f89d4156dba None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.467932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.468568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.468846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.469113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.469799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:10.470134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.471528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.484747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 401/792] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:10.484747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.541521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41808036-d79a-45a7-8bd5-b767ecdc7a57 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.558258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.579980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82be5be-b989-4c07-a9d1-969ce09a7e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.580618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82be5be-b989-4c07-a9d1-969ce09a7e22 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.580888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82be5be-b989-4c07-a9d1-969ce09a7e22 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.581152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82be5be-b989-4c07-a9d1-969ce09a7e22 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.583374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.591849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82be5be-b989-4c07-a9d1-969ce09a7e22 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.593380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82be5be-b989-4c07-a9d1-969ce09a7e22 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.603947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9eb98fd-a7d1-4e1d-8be0-735cbf74264c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-d82be5be-b989-4c07-a9d1-969ce09a7e22 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:10.607878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:10.608781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:10.610256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:10.610256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:10.610256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:10.610256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 392/793] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:10.610480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc38eff-24aa-46bd-9f57-99e48e2abda5 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.610702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc38eff-24aa-46bd-9f57-99e48e2abda5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.610954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cc38eff-24aa-46bd-9f57-99e48e2abda5 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.612310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 393/794] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:10 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:10.617375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc38eff-24aa-46bd-9f57-99e48e2abda5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.645345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.645977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.646301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.646547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.657808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.659319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:10.674301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:10.675226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:10.681845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:10.681845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:10.681845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:10.681845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 402/795] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:10.681845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.726736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cc38eff-24aa-46bd-9f57-99e48e2abda5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.760139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3424d631-7f53-4212-9f51-3adf31407bc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.760845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3424d631-7f53-4212-9f51-3adf31407bc4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.761187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3424d631-7f53-4212-9f51-3adf31407bc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.761519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3424d631-7f53-4212-9f51-3adf31407bc4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.771579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3424d631-7f53-4212-9f51-3adf31407bc4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.773593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3424d631-7f53-4212-9f51-3adf31407bc4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.788535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e1da105-14d4-4fdb-9efb-a02555455a1e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-3424d631-7f53-4212-9f51-3adf31407bc4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:10.788915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:10.789985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:10.791029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:10.791029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:10.791029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:10.792410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 394/796] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:10.793779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719af061-9235-495e-a8b1-af636607a06a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.794282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719af061-9235-495e-a8b1-af636607a06a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.794676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-719af061-9235-495e-a8b1-af636607a06a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.814469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-719af061-9235-495e-a8b1-af636607a06a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:10.815873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-719af061-9235-495e-a8b1-af636607a06a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:10.821066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.827742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:10.828345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:10.828606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:10.828883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:10.829535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:10.829837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:10.830954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:10.831277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:10.845561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 403/797] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:10.846529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.895631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 395/798] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:10 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:10.896479 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719af061-9235-495e-a8b1-af636607a06a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:10.976388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc00459-840d-4715-a4eb-260358527f61 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.020655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.021167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-719af061-9235-495e-a8b1-af636607a06a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.021767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.022402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.022911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.023764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:11.024242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.025804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.038402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 404/799] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:11.039587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.066109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.067068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.067522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.067989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.068836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:11.069473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.071111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.083181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 396/800] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:11.084250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.097284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.111647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.164112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53bfef5-e984-4c67-8b97-6940cdc09cb3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.200311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.200939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.201208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.201475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.209945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.211509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.212230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bf651e8-f6f1-4381-a55b-592beeebc92b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:11.223080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:11.224045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:11.225070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 405/801] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:11.226371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.251439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.252099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.252381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.252659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.260237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.261744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:11.273012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:11.273962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 397/802] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/users/swift => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:11.274934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.343810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff35e14-0021-4f0f-b0f8-774912780c24 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.352524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.360903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.387309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e735caaf-2521-4491-8598-d2de339107db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.387939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735caaf-2521-4491-8598-d2de339107db None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.388213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735caaf-2521-4491-8598-d2de339107db None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.388484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e735caaf-2521-4491-8598-d2de339107db None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.388580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c436a58-edd1-40f6-ab8c-523b5aa51443 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.398675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e735caaf-2521-4491-8598-d2de339107db None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.400761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e735caaf-2521-4491-8598-d2de339107db None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e735caaf-2521-4491-8598-d2de339107db None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:11.415089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:11.416252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 406/803] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:11.417201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e735caaf-2521-4491-8598-d2de339107db None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.429521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.430175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.430452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.430719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.431305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.432721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:11.441387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:11.442554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:11.443528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 398/804] 10.222.0.61 () {56 vars in 1129 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:11.444538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.523804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e735caaf-2521-4491-8598-d2de339107db None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.541797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-290a9f19-b3a2-4d37-9ae0-44d023b55361 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.559651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8737ed6c-974f-4354-82f3-53fa374cb4e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.560374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737ed6c-974f-4354-82f3-53fa374cb4e5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.560644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737ed6c-974f-4354-82f3-53fa374cb4e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.560912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8737ed6c-974f-4354-82f3-53fa374cb4e5 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.561632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8737ed6c-974f-4354-82f3-53fa374cb4e5 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:11.561935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8737ed6c-974f-4354-82f3-53fa374cb4e5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.563390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8737ed6c-974f-4354-82f3-53fa374cb4e5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.572749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.578234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.578841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.579115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.579383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.580105 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:11.580406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.580858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 407/805] 10.222.0.61 () {56 vars in 1149 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:11.581842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.582095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4aab13-31d1-4079-9c84-3d40b8595ed5 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.582372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4aab13-31d1-4079-9c84-3d40b8595ed5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.582636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4aab13-31d1-4079-9c84-3d40b8595ed5 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.586117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 408/806] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:11.586117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4aab13-31d1-4079-9c84-3d40b8595ed5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.601355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 399/807] 10.222.0.61 () {56 vars in 1135 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:11.602145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.615884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.688109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4aab13-31d1-4079-9c84-3d40b8595ed5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.703157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecc69b24-521c-4990-965a-1a79aa3e6e9b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.722876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.723461 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.723732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.724000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.724715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:11.725015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.726584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.736416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51afca18-cdec-48eb-a68f-b2781552b3b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.737013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51afca18-cdec-48eb-a68f-b2781552b3b8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.737282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51afca18-cdec-48eb-a68f-b2781552b3b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.737544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51afca18-cdec-48eb-a68f-b2781552b3b8 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.738193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51afca18-cdec-48eb-a68f-b2781552b3b8 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:11.738486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51afca18-cdec-48eb-a68f-b2781552b3b8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.739833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51afca18-cdec-48eb-a68f-b2781552b3b8 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.744326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 409/808] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:11.744626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 400/809] 10.222.0.61 () {56 vars in 1124 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:11.745759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefd6796-feba-423d-b76f-c2751284d413 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.746060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefd6796-feba-423d-b76f-c2751284d413 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.746331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefd6796-feba-423d-b76f-c2751284d413 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.747291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.763838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cefd6796-feba-423d-b76f-c2751284d413 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:11.765007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cefd6796-feba-423d-b76f-c2751284d413 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:11.813943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.832585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 401/810] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:11 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:11.833394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefd6796-feba-423d-b76f-c2751284d413 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.835447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:11.847483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fffdbb-bddf-4c6f-b387-800908334474 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.883215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.883771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.884040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.884306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.894916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:11.896365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:11.911236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:11.912230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:11.913226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:11.913226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:11.913226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:11.913226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 410/811] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:11.913226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:11.953182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefd6796-feba-423d-b76f-c2751284d413 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:11.988188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:11.988823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:11.989099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:11.989368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:11.999867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.001316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.015475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0be029f-2bfa-4acf-ab39-4796a66450fb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:12.016609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:12.017524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:12.019099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:12.019099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:12.019099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:12.019099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 402/812] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:12.019099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.054369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.056008 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.056707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.057031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.057329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.069358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.120008 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23b8953-a4b4-49f7-9a00-c280a8f2927a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.126914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.128358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.153610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.154476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.154810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.155141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.155751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:12.156126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.157847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.166927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 403/813] 10.222.0.61 () {56 vars in 1124 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:12.166927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.246658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 411/814] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:11 2022] GET /identity/v3/auth/tokens => generated 1002 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:12.247542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.273364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.276689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefe393e-4f4b-479c-8725-b07ae027672a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.295449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.311675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.312307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.312585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.312854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.313492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:12.313793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.315299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.327368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 404/815] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:12.328160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.357440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e07bc8-ad23-45b6-b412-69d7f4a8e387 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.391215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.391814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.392085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.392351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.392985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:12.393279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.394691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.406385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 412/816] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:12.407161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.426650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a087822-465e-4dec-b12d-86919f0c6e30 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.463069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.463737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.464059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.464373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] PATH_INFO: `/v3/services/network` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.464952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.466364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:12.475370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:12.476507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:12.477362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 405/817] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/services/network => generated 88 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:12.478184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.504004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.506670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.509567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0658b365-59fc-492c-94f5-2f42ffab5857 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.545178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.545783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.546130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.546361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.547091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.548462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.573854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 413/818] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:12 2022] POST /identity/v3/endpoints => generated 346 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:12.574731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.580830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2198c6c6-cec2-47d8-b97d-e8e7aad4ca60 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.628106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.628764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.629035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.629301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.641599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.643113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:12.658730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:12.659723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:12.660698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:12.660698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:12.660698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:12.662643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 406/819] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:12.663971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.707560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024bfd74-6ec6-4e95-a9ba-c9ff883a94fb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.741165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.747046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.747693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.747958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.750048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.756318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.757812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:12.770406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:12.771313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:12.772323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 414/820] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:12.777945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.779603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50236e2b-2f80-4d3e-8e3c-7455e3b90bdb None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.797330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.815428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.816434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.816957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.817365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.824401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.826044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:12.838613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:12.839690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:12.840770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:12.842784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 407/821] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:12.843673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.889243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1895d804-6478-46f8-8cba-0af8b22bb869 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.923817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.924344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.924620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.924891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.925527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] RBAC: Adding query filter params (name=network) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:12.925830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.927244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:12.933089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 415/822] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/services?name=network => generated 124 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:12.933886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:12.950463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a41dd64a-f7db-4ead-af75-a938294a2f37 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:12.980634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:12.986747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:12.987367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:12.987646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:12.987900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:12.988549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:12.988849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:12.990290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.002692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 408/823] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.003500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.032697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.054532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a082a182-e303-43c7-999a-670e94e257a9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.089652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.090612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.091067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.091482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.092372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:13.092815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.094433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.106947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-435103c4-8201-42ad-975b-5e492c5a3937 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.111418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 416/824] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.112378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.146551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.147549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.148003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.148560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.149541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:13.149848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.151304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.165086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 409/825] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.166229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.218567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.223411 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4187a-e0e9-44f8-ac1d-921fa015cf20 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.261898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.262534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.262845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.263072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.263746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] RBAC: Adding query filter params (type=network) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:13.264027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.265393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.266571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e915f438-7557-4538-bc51-a6414b20eae4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.271145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 417/826] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/services?type=network => generated 350 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.272090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.279124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.313244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.313888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.314192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.314467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.323809 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.325459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:13.340976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:13.341880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 410/827] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:13.347876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.378565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc150d2-35b9-4710-8280-112a6cd1b051 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.414356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.415010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.415282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.415546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/d505d9008c9540659a3de626874c2457/roles/5c49b35bed6a4d3290bab54c9fa73b9e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.430287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=d505d9008c9540659a3de626874c2457, role_id=5c49b35bed6a4d3290bab54c9fa73b9e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.431747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.447318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 418/828] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:13 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/d505d9008c9540659a3de626874c2457/roles/5c49b35bed6a4d3290bab54c9fa73b9e => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:00:13.448127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.454952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712b4fbe-79d2-467d-a73a-cea149d1b3a6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.471400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.496409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.497266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.497624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.498016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/34b928e9fe314d0fb61122b7a558c1b7/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.500648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=34b928e9fe314d0fb61122b7a558c1b7, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.504526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.532661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 411/829] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:13 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/34b928e9fe314d0fb61122b7a558c1b7/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:00:13.532661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.538333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.598236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8efc582-589f-4e7e-96f4-96b3b7e429cf None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.646620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.649138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.649594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.650861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.651724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.654680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.682755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 419/830] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:13 2022] POST /identity/v3/endpoints => generated 337 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:13.707529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-778481f5-18f1-4163-b196-731f0b52c001 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.754990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7eb9a80-66ca-4f71-a360-42eac722a0af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.755781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb9a80-66ca-4f71-a360-42eac722a0af None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.756285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb9a80-66ca-4f71-a360-42eac722a0af None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.756656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7eb9a80-66ca-4f71-a360-42eac722a0af None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.757490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eb9a80-66ca-4f71-a360-42eac722a0af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:13.757893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eb9a80-66ca-4f71-a360-42eac722a0af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.759528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7eb9a80-66ca-4f71-a360-42eac722a0af None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.775068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 412/831] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:13.781157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:13.917764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b9a8085-7da9-49be-afe4-3d968a4c99b2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:13.929636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:13.953796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e05c213-df7f-4f8d-ba90-92886727e40c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:13.954493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e05c213-df7f-4f8d-ba90-92886727e40c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:13.954849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e05c213-df7f-4f8d-ba90-92886727e40c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:13.955157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e05c213-df7f-4f8d-ba90-92886727e40c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:13.955885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e05c213-df7f-4f8d-ba90-92886727e40c None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=bc807d5f532b4e4c9953ba5e7eb878bb) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:13.956187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e05c213-df7f-4f8d-ba90-92886727e40c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:13.957622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e05c213-df7f-4f8d-ba90-92886727e40c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:13.971150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 420/832] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:13 2022] GET /identity/v3/role_assignments?user.id=bc807d5f532b4e4c9953ba5e7eb878bb&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:14.805559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34286245-2c68-42d4-841d-e7313d550027 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:14.806081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34286245-2c68-42d4-841d-e7313d550027 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:14.806081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34286245-2c68-42d4-841d-e7313d550027 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:14.807579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 413/833] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:14.812952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2c135e-23d7-442a-8364-76e4a764bde6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:14.813348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2c135e-23d7-442a-8364-76e4a764bde6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:14.813511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2c135e-23d7-442a-8364-76e4a764bde6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:14.831852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7e2c135e-23d7-442a-8364-76e4a764bde6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:14.833025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7e2c135e-23d7-442a-8364-76e4a764bde6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:14.900051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 421/834] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:14 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:15.162760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34286245-2c68-42d4-841d-e7313d550027 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:15.191545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:15.268176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34286245-2c68-42d4-841d-e7313d550027 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:15.304030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:15.304660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:15.304926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:15.305188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:15.305766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:15.307167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:15.316325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:15.317350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:15.318238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 414/835] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:15 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:15.323724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2c135e-23d7-442a-8364-76e4a764bde6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:15.427212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2c135e-23d7-442a-8364-76e4a764bde6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:15.430292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:15.461978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b51b09e3-1e5d-49a4-ad11-bc3ce408ef6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:15.462686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b09e3-1e5d-49a4-ad11-bc3ce408ef6f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:15.462963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b09e3-1e5d-49a4-ad11-bc3ce408ef6f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:15.463238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b51b09e3-1e5d-49a4-ad11-bc3ce408ef6f None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:15.464012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51b09e3-1e5d-49a4-ad11-bc3ce408ef6f None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:15.464611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51b09e3-1e5d-49a4-ad11-bc3ce408ef6f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:15.466510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b51b09e3-1e5d-49a4-ad11-bc3ce408ef6f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:15.471444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 422/836] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:15 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:15.476825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:15.575568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:15.586373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119f457a-2747-4b81-a2c7-eef1441211a7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:15.620071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b92efb7-72a0-4bb1-ae82-c2f4c8be2c36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:15.620686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b92efb7-72a0-4bb1-ae82-c2f4c8be2c36 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:15.620960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b92efb7-72a0-4bb1-ae82-c2f4c8be2c36 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:15.621230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b92efb7-72a0-4bb1-ae82-c2f4c8be2c36 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:15.621853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b92efb7-72a0-4bb1-ae82-c2f4c8be2c36 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:15.622261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b92efb7-72a0-4bb1-ae82-c2f4c8be2c36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:15.623639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b92efb7-72a0-4bb1-ae82-c2f4c8be2c36 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:15.628505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 415/837] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:15 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:15.899599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8433b57-1cf2-49b1-8a88-47ef0c8445f8 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:15.900130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8433b57-1cf2-49b1-8a88-47ef0c8445f8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:15.900130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8433b57-1cf2-49b1-8a88-47ef0c8445f8 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:15.901717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 423/838] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:15.907022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee8b29-1a84-4253-8cf1-e516b3b9cf87 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:15.907442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee8b29-1a84-4253-8cf1-e516b3b9cf87 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:15.907618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ee8b29-1a84-4253-8cf1-e516b3b9cf87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:15.926025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c1ee8b29-1a84-4253-8cf1-e516b3b9cf87 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:15.927205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c1ee8b29-1a84-4253-8cf1-e516b3b9cf87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:15.971546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cece5a-b1b1-4424-948f-13fc63c34ae0 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:15.972173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cece5a-b1b1-4424-948f-13fc63c34ae0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:15.972647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96cece5a-b1b1-4424-948f-13fc63c34ae0 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:15.974547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 424/839] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:15.980258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aa6442-a0a6-4463-93b8-5d6c1c9f156e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:15.980815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aa6442-a0a6-4463-93b8-5d6c1c9f156e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:15.981349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97aa6442-a0a6-4463-93b8-5d6c1c9f156e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.000011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 416/840] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:15 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:16.001080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-97aa6442-a0a6-4463-93b8-5d6c1c9f156e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:16.006052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-97aa6442-a0a6-4463-93b8-5d6c1c9f156e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:16.076666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 425/841] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:15 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:16.279179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ee8b29-1a84-4253-8cf1-e516b3b9cf87 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.310737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee35cb3-151b-4212-897e-546d7450da40 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.311107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee35cb3-151b-4212-897e-546d7450da40 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.311277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ee35cb3-151b-4212-897e-546d7450da40 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.313503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 426/842] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:16.319794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fa3378-424e-4020-bf4c-acf770b20313 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.320070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fa3378-424e-4020-bf4c-acf770b20313 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.320341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35fa3378-424e-4020-bf4c-acf770b20313 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.340739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-35fa3378-424e-4020-bf4c-acf770b20313 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:16.342306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-35fa3378-424e-4020-bf4c-acf770b20313 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:16.381703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.399165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ee8b29-1a84-4253-8cf1-e516b3b9cf87 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.439364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.440025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.440299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.440541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.453524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:16.458422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:16.459439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:00:16.460282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:16.462411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 417/843] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/roles/service => generated 85 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:16.463594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.554318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 427/844] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:16 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:16.554318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fa3378-424e-4020-bf4c-acf770b20313 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.575733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8ff00cd-ea6a-4bef-8b03-da95232cc11b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.585270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.611528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.612404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.612865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.613325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.624657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:16.629558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:16.630727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:16.631968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:16.633923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 418/845] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:16.639849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.651630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.685318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35fa3378-424e-4020-bf4c-acf770b20313 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.720851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.721661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.721786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.721974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.722673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:16.723142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:16.724515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:16.739006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 428/846] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:16.739671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.778207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d06edfa-d741-4127-a385-a53a633633b6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.814992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.815604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.815877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.816157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.816808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:16.817108 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:16.818559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:16.832300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 419/847] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:16.834319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.850794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.862128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df168cec-3faf-4ed9-bf8c-2a899ca5414e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.897499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.898209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.898480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.898795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.905886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:16.912286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:16.913729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:16.928154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:16.929467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:16.930401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:16.930401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:16.930401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:16.934334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 429/848] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:16.935036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:16.943884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03088440-faaf-405e-86ca-3958813920f1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:16.979585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:16.980219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:16.980832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:16.981083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:16.992274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:16.993755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:17.008749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:17.009899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:17.011575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:17.011575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:17.011575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:17.011575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 420/849] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:17.011575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.050397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7768265f-d863-4a8a-b7eb-278a68187c87 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.086896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.087450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.087726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.087993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.088576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.089943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:17.099323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:17.100293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:17.101196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 430/850] 10.222.0.61 () {56 vars in 1132 bytes} [Thu Sep 29 02:00:16 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:17.102610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.111630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.120932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab1ca502-e9fd-4572-af4a-06396a50d9bd None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.140720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.154589 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.155230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.155464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.155726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.156357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:17.156654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.158052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.169658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 421/851] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:17.170470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.220023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5075449-e8d3-431c-9b53-ca82337f1e3f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.255285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.255894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.256167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.256433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.257072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:17.257375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.258768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.271277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 431/852] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:17.272200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.281184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3753a19-03c8-4ebc-a34b-900321b57a7f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.316190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.317158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.317603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.318110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.319010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:17.319477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.321126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.330529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 422/853] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:17.331617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.344767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.383487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f01c8590-4a40-4427-bcf9-9e1e6f9dcf67 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.387809 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.422928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.423571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.423952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.424358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.432413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.434185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:17.446968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:17.447953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:17.448899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:17.449428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9080e958-c7a1-4c83-8961-b1c6650acca7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.449503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 432/854] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:17.449503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.487705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.488336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.488604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.488868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.496139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.497578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:17.508967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:17.509881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 423/855] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/users/swift => generated 83 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:17.510890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.569903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255bd91e-745f-4221-aaa6-b1bad5bb315f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.600463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.610904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.611599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.611874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.612135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.612769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:17.613172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.614684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.619685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 433/856] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:17.620447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba394360-9695-4f37-a4d6-ee1964bdd47d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.625147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.631255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.665319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.666194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.667226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.667226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.668196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:17.668594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.670565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.696934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 424/857] 10.222.0.61 () {56 vars in 1149 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:17.698524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.791732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427d8106-116c-40bb-bb1d-05600e43e2c9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.823449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d06b76-5aa3-4ed3-a9e5-a7a73cb0be48 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:17.833455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f3754e-b636-444e-9a6c-49a79c34cc5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.834484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f3754e-b636-444e-9a6c-49a79c34cc5e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.834765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f3754e-b636-444e-9a6c-49a79c34cc5e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.835386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f3754e-b636-444e-9a6c-49a79c34cc5e None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.835874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f3754e-b636-444e-9a6c-49a79c34cc5e None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:17.836182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f3754e-b636-444e-9a6c-49a79c34cc5e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.837626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f3754e-b636-444e-9a6c-49a79c34cc5e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.856852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 434/858] 10.222.0.61 () {56 vars in 1135 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:17.858277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf6c4d-65f5-4431-85e0-e941e6aae99c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.858553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf6c4d-65f5-4431-85e0-e941e6aae99c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.858827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bdf6c4d-65f5-4431-85e0-e941e6aae99c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.860144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 435/859] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:17.865122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdf6c4d-65f5-4431-85e0-e941e6aae99c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:17.865380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159112ec-26d3-4244-b888-2260bd9b027d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:17.866432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159112ec-26d3-4244-b888-2260bd9b027d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:17.866432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159112ec-26d3-4244-b888-2260bd9b027d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:17.866603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159112ec-26d3-4244-b888-2260bd9b027d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:17.898533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.921522 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:17.944606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159112ec-26d3-4244-b888-2260bd9b027d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:17.946094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159112ec-26d3-4244-b888-2260bd9b027d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:17.977777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bdf6c4d-65f5-4431-85e0-e941e6aae99c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.015689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.016300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.016574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.016843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.059416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 425/860] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:18.059416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d8ec53-86e9-4868-b20a-3fa5a1108f3d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.059416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d8ec53-86e9-4868-b20a-3fa5a1108f3d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.059416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9d8ec53-86e9-4868-b20a-3fa5a1108f3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.075207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-d9d8ec53-86e9-4868-b20a-3fa5a1108f3d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:18.076540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d9d8ec53-86e9-4868-b20a-3fa5a1108f3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:18.095405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.096856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.139119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.149636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.158041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 426/861] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:18 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:18.158041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d8ec53-86e9-4868-b20a-3fa5a1108f3d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.221653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 436/862] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:17 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:18.227274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.261579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9d8ec53-86e9-4868-b20a-3fa5a1108f3d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.296853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.297663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.298046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.298571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.309812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.311171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:18.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:18.327890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:18.328860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:18.328860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:18.328860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:18.328860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 427/863] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:18.333606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.356373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722e032a-d9d6-48f2-8a6c-9d4b0396e37b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.392857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.400253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.400939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.401215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.401483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.415170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.416618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.427463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:18.435658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:18.436594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:18.437476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:18.437476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:18.437476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:18.439007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 437/864] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:18.439767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.460498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d437f68-369a-48e5-ba0d-6a1b7e649e7b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.494601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.495497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.495780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.496050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.496692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:18.496993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.498431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.510939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 428/865] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:18.511831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.560628 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb375b1-c02d-4fb9-90ef-5bbf3c81129f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.597426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.598033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.598337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.598602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.599251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.601232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.621364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 438/866] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:18 2022] POST /identity/v3/services => generated 251 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:18.625141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e408dd-3920-428d-aff7-b6a54bb45b3d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.631189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.632145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.697585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.699269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.700028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.700501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.702264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.702563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:18.704075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.706832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.729858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 429/867] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:18.735450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.753190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc2da5b-fea4-4888-9b44-4cb6fab9b9df None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.791572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.792340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.792667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.792999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.800889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.802496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:18.815254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:18.816226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:18.817180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 439/868] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:18.822612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.894055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f8afc0-f801-416c-917d-a51affaa3d64 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.921436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.922111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.922387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.922655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.930457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.931976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:18.942409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5ad4c85-defd-4bd7-87a3-6f2db078032a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:18.943529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:18.944537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:18.945794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 430/869] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:18.951104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:18.966142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.982391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:18.986378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:18.987323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:18.987654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:18.987980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:18.988738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:18.989251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:18.991874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:19.009267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 440/870] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:19.015632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:19.068684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71f513f8-50bd-4a37-bd39-f80aee568880 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:19.108481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:19.109115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:19.109388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:19.109657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:19.110381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:19.110680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:19.112115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:19.128827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 431/871] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:19.135943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:19.145004 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-689b7e19-937a-4893-9f7a-cbfcd28388bc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:19.183484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5e28c0-7ec9-4a7b-8c80-0fc6c417b4ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:19.184091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e28c0-7ec9-4a7b-8c80-0fc6c417b4ce None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:19.184371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e28c0-7ec9-4a7b-8c80-0fc6c417b4ce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:19.184640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5e28c0-7ec9-4a7b-8c80-0fc6c417b4ce None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:19.185360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5e28c0-7ec9-4a7b-8c80-0fc6c417b4ce None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=d505d9008c9540659a3de626874c2457) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:19.185764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5e28c0-7ec9-4a7b-8c80-0fc6c417b4ce None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:19.187274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5e28c0-7ec9-4a7b-8c80-0fc6c417b4ce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:19.194276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 441/872] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:19 2022] GET /identity/v3/role_assignments?user.id=d505d9008c9540659a3de626874c2457&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:19.238144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:19.260933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d440f9fc-de6a-4d38-8053-ed8ed9ae57fd None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:19.299342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e68f96ea-9068-4ea0-bbc5-467fad246802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:19.299967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68f96ea-9068-4ea0-bbc5-467fad246802 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:19.300247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68f96ea-9068-4ea0-bbc5-467fad246802 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:19.300501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e68f96ea-9068-4ea0-bbc5-467fad246802 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:19.301236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68f96ea-9068-4ea0-bbc5-467fad246802 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=34b928e9fe314d0fb61122b7a558c1b7) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:19.301546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68f96ea-9068-4ea0-bbc5-467fad246802 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:19.304177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e68f96ea-9068-4ea0-bbc5-467fad246802 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:19.311299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 432/873] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:19 2022] GET /identity/v3/role_assignments?user.id=34b928e9fe314d0fb61122b7a558c1b7&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:19.857436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93452ba1-cda1-4231-be1a-250e950c63d8 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:19.857834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93452ba1-cda1-4231-be1a-250e950c63d8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:19.858068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93452ba1-cda1-4231-be1a-250e950c63d8 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:19.859618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 442/874] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:19.865029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c465f-8e11-4841-8b39-19cc6073c9ef None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:19.865349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c465f-8e11-4841-8b39-19cc6073c9ef None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:19.865626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca1c465f-8e11-4841-8b39-19cc6073c9ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:19.884210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ca1c465f-8e11-4841-8b39-19cc6073c9ef None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:19.885396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ca1c465f-8e11-4841-8b39-19cc6073c9ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:19.958120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 433/875] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:19 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:20.222268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93452ba1-cda1-4231-be1a-250e950c63d8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:20.281976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:20.340950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93452ba1-cda1-4231-be1a-250e950c63d8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:20.375821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-838a3b2e-5913-49ad-b083-524bb7b7b4b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:20.376429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a3b2e-5913-49ad-b083-524bb7b7b4b8 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:20.376702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a3b2e-5913-49ad-b083-524bb7b7b4b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:20.377011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-838a3b2e-5913-49ad-b083-524bb7b7b4b8 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:20.377687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a3b2e-5913-49ad-b083-524bb7b7b4b8 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:20.379102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-838a3b2e-5913-49ad-b083-524bb7b7b4b8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:20.404517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 443/876] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:20 2022] POST /identity/v3/services => generated 269 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:20.892706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22253f3e-0fa7-4064-b8db-357c004ee82d None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:20.893104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22253f3e-0fa7-4064-b8db-357c004ee82d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:20.893276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22253f3e-0fa7-4064-b8db-357c004ee82d None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:20.894856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 434/877] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:20.903341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f709f04e-2784-44ea-8355-3e7321cd9c02 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:20.903341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f709f04e-2784-44ea-8355-3e7321cd9c02 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:20.903341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f709f04e-2784-44ea-8355-3e7321cd9c02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:20.919700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f709f04e-2784-44ea-8355-3e7321cd9c02 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:20.921163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f709f04e-2784-44ea-8355-3e7321cd9c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:20.999995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 444/878] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:20 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:21.264000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22253f3e-0fa7-4064-b8db-357c004ee82d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:21.344981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:21.372914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22253f3e-0fa7-4064-b8db-357c004ee82d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:21.408613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50203b3e-7dc6-4872-afa6-f1381733d887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:21.409244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50203b3e-7dc6-4872-afa6-f1381733d887 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.409507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50203b3e-7dc6-4872-afa6-f1381733d887 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.409767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50203b3e-7dc6-4872-afa6-f1381733d887 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.410365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50203b3e-7dc6-4872-afa6-f1381733d887 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:21.411731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50203b3e-7dc6-4872-afa6-f1381733d887 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-50203b3e-7dc6-4872-afa6-f1381733d887 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:21.421017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:21.421988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:21.422869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 435/879] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:21 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:21.428908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f709f04e-2784-44ea-8355-3e7321cd9c02 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:21.465798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:21.546683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f709f04e-2784-44ea-8355-3e7321cd9c02 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:21.571486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0775c39c-55e9-4653-bddc-3944238bbfe5 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.571820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0775c39c-55e9-4653-bddc-3944238bbfe5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.572035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0775c39c-55e9-4653-bddc-3944238bbfe5 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.573620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 436/880] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:21.578650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31a7f3a-1434-4c6c-b791-7bea4c08a91c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.578927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31a7f3a-1434-4c6c-b791-7bea4c08a91c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.579196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31a7f3a-1434-4c6c-b791-7bea4c08a91c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.583656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc667a95-336a-452b-befd-205e113f0521 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:21.584289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc667a95-336a-452b-befd-205e113f0521 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.584555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc667a95-336a-452b-befd-205e113f0521 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.584875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc667a95-336a-452b-befd-205e113f0521 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.585516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc667a95-336a-452b-befd-205e113f0521 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:21.585867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc667a95-336a-452b-befd-205e113f0521 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:21.587279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc667a95-336a-452b-befd-205e113f0521 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:21.592442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 445/881] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:21 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:21.593703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50de4f5b-6b99-4e26-b519-b81bd104f9fd None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.593987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50de4f5b-6b99-4e26-b519-b81bd104f9fd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.594346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50de4f5b-6b99-4e26-b519-b81bd104f9fd None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.595843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 446/882] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:21.598369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50de4f5b-6b99-4e26-b519-b81bd104f9fd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:21.599163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f31a7f3a-1434-4c6c-b791-7bea4c08a91c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:21.601199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f31a7f3a-1434-4c6c-b791-7bea4c08a91c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:21.671088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 437/883] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:21 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:21.672750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68549d-22cf-4538-924f-2505fcab9e99 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.672965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68549d-22cf-4538-924f-2505fcab9e99 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.673243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae68549d-22cf-4538-924f-2505fcab9e99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.694980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ae68549d-22cf-4538-924f-2505fcab9e99 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:21.696126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ae68549d-22cf-4538-924f-2505fcab9e99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:21.710253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:21.718548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50de4f5b-6b99-4e26-b519-b81bd104f9fd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:21.755241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:21.756277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.756755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.757304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.758161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:21.758662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:21.760359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:21.763901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:21.766196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 447/884] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:21 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:21.771863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:21.776221 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 438/885] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:21 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:21.874566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf21ecc-b04d-4f0d-8167-8aa1c7575754 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:21.909644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:21.910569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:21.911077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:21.911542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:21.912726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] RBAC: Adding query filter params (interface=public, service_id=784a31e5647a4c79b4f2b61cde962ee9, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:21.913258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:21.914931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:21.920717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 448/886] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:00:21 2022] GET /identity/v3/endpoints?service_id=784a31e5647a4c79b4f2b61cde962ee9&interface=public®ion_id=RegionOne => generated 194 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:21.928605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae68549d-22cf-4538-924f-2505fcab9e99 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:21.961958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:21.987301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:22.054209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae68549d-22cf-4538-924f-2505fcab9e99 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.069399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504596eb-a258-4348-b6c7-10dbd11da09b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.092504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.093118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.093393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.093662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.094323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:22.095702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:22.102151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 439/887] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:21 2022] GET /identity/v3/services => generated 1268 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:00:22.102792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:22.113486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.114575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.115150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.116245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.117058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:22.118700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:22.129240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:22.130548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:22.132086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:22.135005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 449/888] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:21 2022] GET /identity/v3/services/object-store => generated 93 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:22.147805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:22.184199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:22.234783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43052470-1e81-4aee-b8a7-db6899a50f9e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.274454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.275137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.275375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.275555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.281467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:22.292856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:22.300910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:22.301993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:22.304010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 440/889] 10.222.0.61 () {56 vars in 1120 bytes} [Thu Sep 29 02:00:22 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:22.308008 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a69b6f25-66a8-4817-9e51-14e50591699f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.310271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:22.353192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.354184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.354625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.355033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.355809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:22.356104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:22.357468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:22.363758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 450/890] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:22 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:22.369777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:22.441829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6955108c-6f69-49a5-bc3e-ba423a424a2a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.486507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.487286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-306d0944-fa83-40e5-b03f-46a902e651df None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.488156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.488639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.489412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.491504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:22.492102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:22.493770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:22.502274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:22.507391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 441/891] 10.222.0.61 () {56 vars in 1115 bytes} [Thu Sep 29 02:00:22 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:22.515280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:22.546027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d7503ee-f66f-4233-887e-de2da5159804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.546027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7503ee-f66f-4233-887e-de2da5159804 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.546027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7503ee-f66f-4233-887e-de2da5159804 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.546027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d7503ee-f66f-4233-887e-de2da5159804 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.546483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7503ee-f66f-4233-887e-de2da5159804 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:22.546483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7503ee-f66f-4233-887e-de2da5159804 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:22.548249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d7503ee-f66f-4233-887e-de2da5159804 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:22.558144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 451/892] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:22 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:22.616212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fc9de-edc3-4043-8d64-6695f7248c15 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.616578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fc9de-edc3-4043-8d64-6695f7248c15 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.616749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86fc9de-edc3-4043-8d64-6695f7248c15 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.622443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 452/893] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:22 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:22.624235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36b3a0-d099-40e0-9c51-5c18b3e4dc96 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.624578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36b3a0-d099-40e0-9c51-5c18b3e4dc96 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.624856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed36b3a0-d099-40e0-9c51-5c18b3e4dc96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.644484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ed36b3a0-d099-40e0-9c51-5c18b3e4dc96 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:22.645918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ed36b3a0-d099-40e0-9c51-5c18b3e4dc96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:22.647679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03084b90-3c11-42af-91b9-c8fc3e19c89e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.678771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:22.695053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.695960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.696403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.696810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.711841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:22.713865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:22.738041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 453/894] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:22 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:22.738281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:22.739160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:22.740132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:22.740132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:22.740132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:22.740132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 442/895] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 225 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:22.745484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36b3a0-d099-40e0-9c51-5c18b3e4dc96 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:22.855811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed36b3a0-d099-40e0-9c51-5c18b3e4dc96 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:22.891137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:22.892142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:22.892621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:22.893101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:22.893958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:22.894508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:22.896153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:22.908619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 454/896] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:22.914826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:22.958408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.022328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.025217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5060ec3-a604-4602-88e3-10c37cd3b001 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.026656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.071889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.073100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.074149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.074380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.085167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.086794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:23.099466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:23.100532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:23.101510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 443/897] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:00:22 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:23.109626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.146044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d89acfa-3473-4b7e-9e70-826deeb6d3ea None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.190792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.191988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.192256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.192648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.193485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.195146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:23.206638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:23.207718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:23.208861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:23.213491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 455/898] 10.222.0.61 () {56 vars in 1132 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:23.214478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.226841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.235315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4940731c-dbf0-4b7a-980b-86e8b8af0cdd None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.272152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.273009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.273383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.273749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.274623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:23.275183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.276648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.300062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 444/899] 10.222.0.61 () {56 vars in 1149 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:23.301073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.302653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.347897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1823100-ba20-4a22-b1ce-c0f153d51c4a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.394173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.394791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.395059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.395321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.395990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:23.396495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.399008 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.406199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 456/900] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:23.411052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.432270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcc0394b-0304-47a5-b2ac-41f7dae2e25e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.474038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.475021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.475518 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.475974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.502052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.530199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff587f56-0cd9-473a-92c0-91fb947a2dae None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.553659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.555277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.569309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.575016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.575786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.576272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.576704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.577498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:23.577950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.579712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.585594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 457/901] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:23.590963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.672937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 445/902] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:23.674074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.699849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a35758-471d-490a-8017-f754825eb275 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.732199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.738367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.739301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.739881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.740355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.741246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] RBAC: Adding query filter params (interface=public, service_id=1edbe96276104339b2c2310f2cc62ab8, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:23.741753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.743341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.758329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 458/903] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/endpoints?service_id=1edbe96276104339b2c2310f2cc62ab8&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:23.759715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.787601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b90ef6e3-4f7b-4a0e-af30-57a9102adabc None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.827665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:23.832996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.833886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.834348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.834765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.848683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.850846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:23.872383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:23.873366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:23.874274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:23.874274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:23.874274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:23.879806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 446/904] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:23.880919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:23.883185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06b38ba1-3d44-4e77-9e5a-54b19d72d448 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:23.922602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:23.923231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:23.923502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:23.923768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:23.924372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:23.925905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:23.938240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 459/905] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/services => generated 1268 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:00:24.011132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e51d4ce4-ff63-4e95-9a65-26c649c5909b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:24.019560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:24.050737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:24.051832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.052552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.053052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.054286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:24.054806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:24.056422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:24.070404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 447/906] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:24.076581 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:24.196349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071065cc-61c0-48cf-b316-462487b5996d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:24.220233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:24.232195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c54098-87f5-427f-88d6-e19efac0c6a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:24.233196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c54098-87f5-427f-88d6-e19efac0c6a2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.233621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c54098-87f5-427f-88d6-e19efac0c6a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.234070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c54098-87f5-427f-88d6-e19efac0c6a2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.241515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c54098-87f5-427f-88d6-e19efac0c6a2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:24.243190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c54098-87f5-427f-88d6-e19efac0c6a2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b2c54098-87f5-427f-88d6-e19efac0c6a2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:24.254950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:24.256215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:24.257317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:24.259065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 460/907] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:24.266090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:24.377408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdfdf48d-4db2-4193-9520-f59d86a85d5d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:24.413600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:24.414493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.414608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb354d39-1daf-4bed-a967-caa700dedf96 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.414836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb354d39-1daf-4bed-a967-caa700dedf96 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.415115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb354d39-1daf-4bed-a967-caa700dedf96 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.415317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.415714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.416595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:24.417038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:24.417747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 461/908] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:24.419036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:24.421719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f2e83-1170-40c0-a5cf-32865b49e76c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.421989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f2e83-1170-40c0-a5cf-32865b49e76c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.422285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7f2e83-1170-40c0-a5cf-32865b49e76c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.433635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 448/909] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:24.440349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:24.444274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ad7f2e83-1170-40c0-a5cf-32865b49e76c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:24.445464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ad7f2e83-1170-40c0-a5cf-32865b49e76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:24.471022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:24.514060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 462/910] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:24 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:24.553762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c8b4433-d986-4c6f-bc93-c49461b1389c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:24.592780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b9cc1d-8930-480f-b633-a85c4b287bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:24.593392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9cc1d-8930-480f-b633-a85c4b287bb0 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.593671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9cc1d-8930-480f-b633-a85c4b287bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.593944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b9cc1d-8930-480f-b633-a85c4b287bb0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.594701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9cc1d-8930-480f-b633-a85c4b287bb0 None admin] RBAC: Adding query filter params (role.id=3495b485e650453790bdb49f328d4305, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=d505d9008c9540659a3de626874c2457) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:24.595004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9cc1d-8930-480f-b633-a85c4b287bb0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:24.596448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b9cc1d-8930-480f-b633-a85c4b287bb0 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:24.602763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 449/911] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:24 2022] GET /identity/v3/role_assignments?user.id=d505d9008c9540659a3de626874c2457&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=3495b485e650453790bdb49f328d4305 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:24.790103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f2e83-1170-40c0-a5cf-32865b49e76c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:24.827729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7956e6b9-a498-4b18-adf3-d9f955e7d6a5 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.828880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7956e6b9-a498-4b18-adf3-d9f955e7d6a5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.829325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7956e6b9-a498-4b18-adf3-d9f955e7d6a5 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.831052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 450/912] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:24.840173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2930c7df-7218-430f-93b0-16ba61562465 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.840478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2930c7df-7218-430f-93b0-16ba61562465 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.840704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2930c7df-7218-430f-93b0-16ba61562465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.861808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2930c7df-7218-430f-93b0-16ba61562465 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:24.863294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2930c7df-7218-430f-93b0-16ba61562465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:24.902562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7f2e83-1170-40c0-a5cf-32865b49e76c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:24.918687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:24.944634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:24.945822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:24.946239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:24.946601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:24.947406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:24.948895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:24.958042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 451/913] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:24 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:24.963793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:24.964833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:24.965688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 463/914] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:24 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:24.971004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2930c7df-7218-430f-93b0-16ba61562465 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:24.993352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:25.079865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2930c7df-7218-430f-93b0-16ba61562465 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:25.118526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:25.119525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.119990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.120443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.121292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:25.121793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:25.123389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:25.128485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 452/915] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:24 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:25.134167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:25.234258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c11fd92f-f316-47bf-9c34-71b6e6c89b46 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:25.260984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:25.270990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:25.271610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.271887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.272158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.272789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:25.273208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:25.274919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:25.282049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 464/916] 10.222.0.61 () {56 vars in 1114 bytes} [Thu Sep 29 02:00:25 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:25.285577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:25.334497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:25.381737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:25.401586 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce3db99-f432-4049-9c30-f684473afb92 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:25.404018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5a11e4d-8b85-4cca-94e3-dd16ab7bc3a8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:25.440927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa202c1-8262-4324-b2c8-f6b62c42f049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:25.441844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa202c1-8262-4324-b2c8-f6b62c42f049 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.442353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa202c1-8262-4324-b2c8-f6b62c42f049 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.443107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa202c1-8262-4324-b2c8-f6b62c42f049 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.443748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7720ba69-179a-492b-9f5f-a8c8d04dab29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:25.444172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa202c1-8262-4324-b2c8-f6b62c42f049 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:25.444359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7720ba69-179a-492b-9f5f-a8c8d04dab29 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:25.444633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7720ba69-179a-492b-9f5f-a8c8d04dab29 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:25.444894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7720ba69-179a-492b-9f5f-a8c8d04dab29 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:25.445548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7720ba69-179a-492b-9f5f-a8c8d04dab29 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:25.446512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa202c1-8262-4324-b2c8-f6b62c42f049 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:25.446934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7720ba69-179a-492b-9f5f-a8c8d04dab29 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:25.470895 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 453/917] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:25 2022] POST /identity/v3/services => generated 241 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:00:25.479565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 465/918] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:25 2022] POST /identity/v3/endpoints => generated 346 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:00:26.241950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c5039e-168d-4291-8c97-a6687f6bc3ae None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:26.242335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c5039e-168d-4291-8c97-a6687f6bc3ae None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:26.242538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c5039e-168d-4291-8c97-a6687f6bc3ae None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:26.244133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 454/919] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:26.249585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89d41c3-e418-4589-979a-71ac5c9fac25 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:26.250091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89d41c3-e418-4589-979a-71ac5c9fac25 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:26.250191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89d41c3-e418-4589-979a-71ac5c9fac25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:26.269141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b89d41c3-e418-4589-979a-71ac5c9fac25 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:26.270352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b89d41c3-e418-4589-979a-71ac5c9fac25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:26.345774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 466/920] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:26.609797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c5039e-168d-4291-8c97-a6687f6bc3ae None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:26.713897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c5039e-168d-4291-8c97-a6687f6bc3ae None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:26.751677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3013c0ce-8039-45e0-b59c-c3bb9ef975ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:26.752316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3013c0ce-8039-45e0-b59c-c3bb9ef975ac None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:26.752589 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3013c0ce-8039-45e0-b59c-c3bb9ef975ac None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:26.752856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3013c0ce-8039-45e0-b59c-c3bb9ef975ac None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:26.753443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3013c0ce-8039-45e0-b59c-c3bb9ef975ac None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:26.754853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3013c0ce-8039-45e0-b59c-c3bb9ef975ac None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-3013c0ce-8039-45e0-b59c-c3bb9ef975ac None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:26.764059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:26.765040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:26.765925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 455/921] 10.222.0.61 () {56 vars in 1132 bytes} [Thu Sep 29 02:00:26 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:26.771539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89d41c3-e418-4589-979a-71ac5c9fac25 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:26.797046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:26.833716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46898bd0-45cf-43f9-bbc9-5e57ab9ecac7 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:26.834092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46898bd0-45cf-43f9-bbc9-5e57ab9ecac7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:26.834280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46898bd0-45cf-43f9-bbc9-5e57ab9ecac7 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:26.835755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 456/922] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:26.841059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86287ed-9c4e-41e3-a7f8-1f8b228bdde9 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:26.841338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86287ed-9c4e-41e3-a7f8-1f8b228bdde9 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:26.841615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c86287ed-9c4e-41e3-a7f8-1f8b228bdde9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:26.860392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c86287ed-9c4e-41e3-a7f8-1f8b228bdde9 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:26.861587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c86287ed-9c4e-41e3-a7f8-1f8b228bdde9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:26.864609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:26.878589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89d41c3-e418-4589-979a-71ac5c9fac25 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:26.917433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:26.918177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:26.918616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:26.918997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:26.919704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:26.920080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:26.921734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:26.927158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 467/923] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:26 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:26.934647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:26.944478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 457/924] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:26 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:27.022409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:27.033944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41502977-1e8e-4c93-8699-904ea627734d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:27.068167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:27.068772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.069148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.069415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.070058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:27.070358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:27.071718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:27.076958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 468/925] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:26 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:27.082645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86287ed-9c4e-41e3-a7f8-1f8b228bdde9 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:27.187673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c86287ed-9c4e-41e3-a7f8-1f8b228bdde9 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:27.217715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:27.230904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:27.231647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.231920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.232205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.233011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:27.234588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:27.246481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:27.265457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 458/926] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:27 2022] POST /identity/v3/endpoints => generated 347 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:27.338484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f2c056b-8259-4a16-aa22-0c60a7c48c7e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:27.384764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:27.385837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.386342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.386813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.399312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:27.404841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:27.405876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:27.406822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:27.414200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 469/927] 10.222.0.61 () {56 vars in 1120 bytes} [Thu Sep 29 02:00:27 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:27.416712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:27.524154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed826e62-78ef-4106-b0a8-12156c5ae552 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:27.565341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a0af2bd-1393-4521-8710-203d2ada74c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:27.566031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0af2bd-1393-4521-8710-203d2ada74c5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.566340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0af2bd-1393-4521-8710-203d2ada74c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.566612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0af2bd-1393-4521-8710-203d2ada74c5 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.567297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0af2bd-1393-4521-8710-203d2ada74c5 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:27.567657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0af2bd-1393-4521-8710-203d2ada74c5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:27.569098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a0af2bd-1393-4521-8710-203d2ada74c5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:27.582213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 459/928] 10.222.0.61 () {56 vars in 1115 bytes} [Thu Sep 29 02:00:27 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:27.585780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:27.600756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:27.698127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ed4e2c-7bf0-42c2-8ce4-edb1b399147e None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.698702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ed4e2c-7bf0-42c2-8ce4-edb1b399147e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.698925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ed4e2c-7bf0-42c2-8ce4-edb1b399147e None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.699627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acca2ac4-8daf-4294-bf9a-aa6584f1c278 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:27.700888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 460/929] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:27 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:27.707994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d59492-5ded-43a7-91c5-345e83f48024 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.708499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d59492-5ded-43a7-91c5-345e83f48024 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.708866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8d59492-5ded-43a7-91c5-345e83f48024 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.729307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-e8d59492-5ded-43a7-91c5-345e83f48024 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:27.730803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e8d59492-5ded-43a7-91c5-345e83f48024 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:27.738720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:27.739356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.739722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.739994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.750528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:27.751976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:27.767313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:27.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:27.769679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:27.769679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:27.769679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:27.769679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 470/930] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:27.779430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:27.790074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:27.813052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 461/931] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:27 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:27.814417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320406fc-137e-4c2c-b451-22baabd67240 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.814756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320406fc-137e-4c2c-b451-22baabd67240 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.815028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320406fc-137e-4c2c-b451-22baabd67240 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.816567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 462/932] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:27.823540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0496533f-50bc-4353-9a3d-ca5ca9428afc None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.823934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0496533f-50bc-4353-9a3d-ca5ca9428afc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.824305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0496533f-50bc-4353-9a3d-ca5ca9428afc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.849272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0496533f-50bc-4353-9a3d-ca5ca9428afc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:27.851610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:27.852257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0496533f-50bc-4353-9a3d-ca5ca9428afc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:27.898890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ec2cfb-af9e-4de6-b25f-bb9ebf2555b6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:27.943506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:27.944384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:27.944797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:27.945223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:27.945994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:27.946477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:27.947979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:27.950123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 463/933] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:27 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:27.966372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 471/934] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:27.972091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0496533f-50bc-4353-9a3d-ca5ca9428afc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.074344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0496533f-50bc-4353-9a3d-ca5ca9428afc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.086997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.097991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.124077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.125182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.125674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.126130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.136891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.138387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:28.151214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:28.152200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:28.153632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:28.157002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 464/935] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:00:27 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:28.158031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.216729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc28d92d-e9cb-4cc8-a272-738b1df3c64d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.248314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.266361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.267005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.267200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed304f1-b742-4fdd-bf96-54e85e980867 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.267303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.267527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.268104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.269472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:28.278473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:28.279529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.281183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 472/936] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/services/object-store => generated 93 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:28.281994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.311443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.311984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.312260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.312531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.313257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:28.313561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.315037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.337823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 465/937] 10.222.0.61 () {56 vars in 1149 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:28.337823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.371619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.392283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cadd2a-e5d2-4c94-859b-408e153425ac None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.425686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.426467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.427070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.427646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.428436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.430319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.432952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING keystone.server.flask.common [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 29 02:00:28.442368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80b99ee0-b124-4177-b56a-fdee02bc223d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.452490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.sql.core [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 29 02:00:28.452490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 29 02:00:28.452490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [parameters: {'id': '2b036e8cb4c542e9bf733135c5c47595', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 29 02:00:28.452490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81684) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2b036e8cb4c542e9bf733135c5c47595.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2b036e8cb4c542e9bf733135c5c47595. Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application conn.query(q) Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 02:00:28.470175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result.read() Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 29 02:00:28.471093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application next(self.gen) Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application yield resource Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application next(self.gen) Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.session.rollback() Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.force_reraise() Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise self.value Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application session.commit() Sep 29 02:00:28.472113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._prepare_impl() Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.session.flush() Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._flush(objects) Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application compat.raise_( Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application flush_context.execute() Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 29 02:00:28.473620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rec.execute(self) Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 29 02:00:28.475293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application conn.query(q) Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application result.read() Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 02:00:28.476113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [parameters: {'id': '2b036e8cb4c542e9bf733135c5c47595', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.478730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:28.481011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 2b036e8cb4c542e9bf733135c5c47595. Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 473/938] 10.222.0.61 () {60 vars in 1140 bytes} [Thu Sep 29 02:00:28 2022] POST /identity/v3/projects => generated 262 bytes in 191 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:28.482159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.490482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.491109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.491389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.491661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.492299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:28.492600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.494077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.499935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 466/939] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:28.500798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.526833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.597040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eb0cd39-7fa9-487a-b23a-ca09f13391ba None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.606711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.611581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced98148-f56f-4c4b-813a-9a87065598c4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.635763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.636398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.636676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.636947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.648360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.650144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.653700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.654368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.654640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.654903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.662986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.664424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:28.664927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:28.666045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:28.672059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:28.672059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:28.672059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:28.672059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 474/940] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:28.672059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:28.683729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:28.684764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 467/941] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:28.685896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.775111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.784713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd9ff485-16d4-4088-bb22-b7ce7d4dc3ce None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.791728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816e8062-6566-44f9-9294-4f8447ea4f26 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.820370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.820976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.821253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.821520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.822177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:28.822480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.823846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.828115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.828722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.829004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.829274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.829909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:28.830240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.836652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 475/942] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:28.836808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.836904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.853084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 468/943] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:28.853084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:28.871849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:28.947190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d12ed4-b548-4eae-bee9-6dfc265b8752 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.959447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-982becaf-f52d-4a8d-91a5-b6761186a2f1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:28.982782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.983576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.983966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.984243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.984919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:28.985223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:28.986735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:28.998096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:28.998713 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:28.998985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:28.999253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:28.999947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] RBAC: Adding query filter params (interface=public, service_id=bb0537d91e6a42f5a3f7ec33c184f688, region_id=RegionOne) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:29.000251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:29.001695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:29.004525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 476/944] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:29.005253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:29.023817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 469/945] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:00:28 2022] GET /identity/v3/endpoints?service_id=bb0537d91e6a42f5a3f7ec33c184f688&interface=public®ion_id=RegionOne => generated 194 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:29.027297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:29.033483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:29.119595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:29.145884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4542375-dd44-465e-afb9-c27ad80ecd7d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:29.159955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce773b27-b10a-47dd-b3ac-04b737985770 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:29.187338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:29.187962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:29.188238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:29.188569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:29.189184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:29.190617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:29.197985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 470/946] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:29 2022] GET /identity/v3/services => generated 1497 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:00:29.208420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:29.209260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:29.209636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:29.210034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:29.220522 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:29.222210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:29.235425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:29.236566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:29.237707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:29.242124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 477/947] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:29 2022] GET /identity/v3/projects/service => generated 88 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:29.245729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:29.365894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3568a26-8e4c-4bcb-a248-1f6e184b2531 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:29.408313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afbbc862-180f-4bc6-8300-e873ad0b19d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:29.409100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbbc862-180f-4bc6-8300-e873ad0b19d6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:29.409477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbbc862-180f-4bc6-8300-e873ad0b19d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:29.409846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afbbc862-180f-4bc6-8300-e873ad0b19d6 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:29.411161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbbc862-180f-4bc6-8300-e873ad0b19d6 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:29.411161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbbc862-180f-4bc6-8300-e873ad0b19d6 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:29.413043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afbbc862-180f-4bc6-8300-e873ad0b19d6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:29.429369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 471/948] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:29 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:29.434573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:29.521522 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:29.527199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf87f32-21f0-46ab-881b-b959efcb9b19 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:29.527753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf87f32-21f0-46ab-881b-b959efcb9b19 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:29.528419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccf87f32-21f0-46ab-881b-b959efcb9b19 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:29.531217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 472/949] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:29 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:29.537421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68499241-865a-49e9-b64e-e19732ead84a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:29.537990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68499241-865a-49e9-b64e-e19732ead84a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:29.538493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68499241-865a-49e9-b64e-e19732ead84a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:29.550990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-419fca16-957e-457a-9169-83e7f6bb5435 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:29.554026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:29.560938 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-68499241-865a-49e9-b64e-e19732ead84a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:29.562480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-68499241-865a-49e9-b64e-e19732ead84a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:29.586762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:29.587421 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:29.587691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:29.588037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/d505d9008c9540659a3de626874c2457/roles/3495b485e650453790bdb49f328d4305` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:29.590223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=d505d9008c9540659a3de626874c2457, role_id=3495b485e650453790bdb49f328d4305)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:29.591643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:29.609851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 478/950] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:29 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/d505d9008c9540659a3de626874c2457/roles/3495b485e650453790bdb49f328d4305 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:00:29.646766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 473/951] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:29 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:29.916167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:30.021363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc6b5161-f36d-4b2e-b3b0-169a7e9c2bf0 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:30.056226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:30.056952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:30.057286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:30.057617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:30.058293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:30.059730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:30.062886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:30.070264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:30.071582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:30.072785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:30.073813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 479/952] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:29 2022] GET /identity/v3/services/compute => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:30.081944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68499241-865a-49e9-b64e-e19732ead84a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:30.191082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68499241-865a-49e9-b64e-e19732ead84a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:30.226450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b97af0-c798-4a09-a4f7-ab924e1d0cfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:30.227205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b97af0-c798-4a09-a4f7-ab924e1d0cfc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:30.227587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b97af0-c798-4a09-a4f7-ab924e1d0cfc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:30.227948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b97af0-c798-4a09-a4f7-ab924e1d0cfc None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:30.228670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b97af0-c798-4a09-a4f7-ab924e1d0cfc None admin] RBAC: Adding query filter params (name=compute) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:30.229068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b97af0-c798-4a09-a4f7-ab924e1d0cfc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:30.230561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b97af0-c798-4a09-a4f7-ab924e1d0cfc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:30.232418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:30.236225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 474/953] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:30 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:30.243099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:30.346787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d0fe68-8c83-4d50-a4e9-d6447ec68112 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:30.382198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65594ab9-856a-4065-92cd-dd31f1ec3329 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:30.382808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65594ab9-856a-4065-92cd-dd31f1ec3329 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:30.383076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65594ab9-856a-4065-92cd-dd31f1ec3329 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:30.383339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65594ab9-856a-4065-92cd-dd31f1ec3329 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:30.383971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65594ab9-856a-4065-92cd-dd31f1ec3329 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:30.384266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65594ab9-856a-4065-92cd-dd31f1ec3329 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:30.385623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65594ab9-856a-4065-92cd-dd31f1ec3329 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:30.390661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 480/954] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:30 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:31.398691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc1a0ff-73a2-4f03-863c-a2bea102e3b9 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.399230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc1a0ff-73a2-4f03-863c-a2bea102e3b9 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.399230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc1a0ff-73a2-4f03-863c-a2bea102e3b9 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.400791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 475/955] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:31.406458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c8f025-79f4-444b-876a-83296ce4d068 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.406853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c8f025-79f4-444b-876a-83296ce4d068 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.407017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c8f025-79f4-444b-876a-83296ce4d068 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.426384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d7c8f025-79f4-444b-876a-83296ce4d068 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:31.428816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d7c8f025-79f4-444b-876a-83296ce4d068 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:31.488904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:31.498234 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 481/956] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:31.503268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d91613-a41c-4876-8e2c-6d7ad3a5ed16 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.503573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d91613-a41c-4876-8e2c-6d7ad3a5ed16 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.503847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d91613-a41c-4876-8e2c-6d7ad3a5ed16 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.505583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 476/957] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:31.510877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a549b6-558f-4b04-9acd-dd573808919d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.511158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a549b6-558f-4b04-9acd-dd573808919d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.511427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a549b6-558f-4b04-9acd-dd573808919d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.530359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-45a549b6-558f-4b04-9acd-dd573808919d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:31.531553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-45a549b6-558f-4b04-9acd-dd573808919d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:31.598490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 482/958] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:31.790578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d91613-a41c-4876-8e2c-6d7ad3a5ed16 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:31.793978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4caa570-5ab6-4dda-a1f2-2efb2413f7bd None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.794295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4caa570-5ab6-4dda-a1f2-2efb2413f7bd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.794566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4caa570-5ab6-4dda-a1f2-2efb2413f7bd None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.796047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 483/959] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:31.801182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b7e352-fb4b-4c6a-a186-f464ef3d5222 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.801464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b7e352-fb4b-4c6a-a186-f464ef3d5222 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.801790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b7e352-fb4b-4c6a-a186-f464ef3d5222 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.821318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e8b7e352-fb4b-4c6a-a186-f464ef3d5222 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:31.822549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e8b7e352-fb4b-4c6a-a186-f464ef3d5222 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:31.892709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 484/960] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:31 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:31.899081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b7e352-fb4b-4c6a-a186-f464ef3d5222 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:31.909834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d91613-a41c-4876-8e2c-6d7ad3a5ed16 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:31.951903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:31.952741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:31.953068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:31.953334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:31.954161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:31.955755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:31.958648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING keystone.server.flask.common [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 29 02:00:31.961800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:31.977153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.sql.core [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 29 02:00:31.977153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 29 02:00:31.977153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [parameters: {'user_id': '88380b0f895d4472bb9d19961744dcd5', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 29 02:00:31.977153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81683) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application conn.query(q) Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result.read() Sep 29 02:00:31.986668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 02:00:31.987667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application next(self.gen) Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application yield resource Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application next(self.gen) Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.session.rollback() Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.force_reraise() Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise self.value Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application session.commit() Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 29 02:00:31.994580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._prepare_impl() Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.session.flush() Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._flush(objects) Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application compat.raise_( Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application flush_context.execute() Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rec.execute(self) Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 29 02:00:31.995491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 29 02:00:31.996401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result = self._query(query) Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application conn.query(q) Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application result.read() Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 29 02:00:31.997258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [parameters: {'user_id': '88380b0f895d4472bb9d19961744dcd5', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:31.999537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 29 02:00:32.001428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 477/961] 10.222.0.61 () {60 vars in 1131 bytes} [Thu Sep 29 02:00:31 2022] POST /identity/v3/users => generated 160 bytes in 198 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:32.003651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.024464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b7e352-fb4b-4c6a-a186-f464ef3d5222 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.061783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.062955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.063426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.063888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.064664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.066304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:32.076328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:32.077342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:32.078218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:32.078987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.081199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 485/962] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:31 2022] GET /identity/v3/services/object-store => generated 93 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:32.087088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.132837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f78fb8-ca52-4887-acf3-17c62e5d2804 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.171598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.172253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.172527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.172791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.181687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.183321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:32.196016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:32.197007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 478/963] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:31 2022] GET /identity/v3/users/cinder => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:32.198027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.201363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e220664-e3aa-44a0-b7a9-34864625f7ae None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.219546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.237745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.238386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.238660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.238935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.239837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:32.240145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.241525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.246542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 486/964] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:32.247238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.316289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-063ed372-c9a9-46d8-b3e4-94aab8ff28d4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.340128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.352545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.353227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.353503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.353772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.359537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcfd136-79b3-427e-b13e-5dbc8a8cbcb2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.363956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:32.369959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:32.371000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 479/965] 10.222.0.61 () {56 vars in 1120 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:32.375064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.404816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.405410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.405684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.405953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.406682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:32.407246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.408436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.445757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 487/966] 10.222.0.61 () {56 vars in 1101 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:32.446656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.493664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245a4550-b458-4915-b774-e600ecfcbae7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.506905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.547884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.548545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.548818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.549117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.549781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:32.550122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.552063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.566044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 480/967] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:32.571049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.595069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c68b1e4-8273-4f67-8e9c-1b6f22fbe2b9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.628100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.639727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbdd7508-f1ef-4ae2-99ea-4005d874682e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.640347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdd7508-f1ef-4ae2-99ea-4005d874682e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.640619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdd7508-f1ef-4ae2-99ea-4005d874682e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.640885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbdd7508-f1ef-4ae2-99ea-4005d874682e None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.641735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdd7508-f1ef-4ae2-99ea-4005d874682e None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:32.642420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdd7508-f1ef-4ae2-99ea-4005d874682e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.644141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbdd7508-f1ef-4ae2-99ea-4005d874682e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.659521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 488/968] 10.222.0.61 () {56 vars in 1115 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:32.659521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb0486d-0731-46b4-a3df-c67dfc2ee1bf None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.659521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb0486d-0731-46b4-a3df-c67dfc2ee1bf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.659521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb0486d-0731-46b4-a3df-c67dfc2ee1bf None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.659521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 489/969] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:32.660402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb0486d-0731-46b4-a3df-c67dfc2ee1bf None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.692232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8bb83df-59ae-4bf8-8430-3e88a9033d9e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.733863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6a1e14-a10d-4cfd-9ab0-26ee6bad52a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.734581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a1e14-a10d-4cfd-9ab0-26ee6bad52a9 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.734856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a1e14-a10d-4cfd-9ab0-26ee6bad52a9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.735132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a1e14-a10d-4cfd-9ab0-26ee6bad52a9 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.735850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6a1e14-a10d-4cfd-9ab0-26ee6bad52a9 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.737277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af6a1e14-a10d-4cfd-9ab0-26ee6bad52a9 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.763101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 481/970] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:32 2022] POST /identity/v3/endpoints => generated 349 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:00:32.765514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d49de69-4c6f-4304-9e9c-9a2bdc81373d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.766159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d49de69-4c6f-4304-9e9c-9a2bdc81373d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.766798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d49de69-4c6f-4304-9e9c-9a2bdc81373d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.785851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.789939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-4d49de69-4c6f-4304-9e9c-9a2bdc81373d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:32.791630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4d49de69-4c6f-4304-9e9c-9a2bdc81373d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:32.793354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb0486d-0731-46b4-a3df-c67dfc2ee1bf None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:32.837385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:32.837934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:32.838234 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:32.838500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:32.850488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:32.851952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:32.866080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 482/971] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:32 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:32.868354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:32.869313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:32.870269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:32.870269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:32.870269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:32.873388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 490/972] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:32.875318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d49de69-4c6f-4304-9e9c-9a2bdc81373d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:32.909047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:32.983532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d49de69-4c6f-4304-9e9c-9a2bdc81373d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:33.017819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:33.018461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:33.018736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:33.019055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:33.019708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:33.020005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:33.021535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:33.037594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 483/973] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:33.039015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:33.128265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:33.142955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8dd84-b8dd-4228-ac24-532df4b6a72e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:33.182507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:33.183145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:33.183419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:33.183690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:33.193485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:33.195808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:33.209033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:33.210246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 29 02:00:33.211434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:33.212544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 491/974] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:00:33 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:33.217994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:33.231737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:33.233626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:33.241501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29bee6e-3526-4285-9bdc-b0d958b1108a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:33.278472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:33.279084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:33.279351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:33.279615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:33.280248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:33.281635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:33.297388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 484/975] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:33 2022] POST /identity/v3/services => generated 242 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:33.341059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcd75cb4-8466-4c45-93bb-25e3b27f2c1a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:33.389026 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:33.389659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:33.389933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:33.390232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:33.391012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:33.391318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:33.392831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:33.414141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 492/976] 10.222.0.61 () {56 vars in 1149 bytes} [Thu Sep 29 02:00:33 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:33.419791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:33.531510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8872ed50-96d0-4eeb-a826-f1982b84387d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:33.572481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:33.573093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:33.573366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:33.573626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:33.595951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:33.647164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:33.648542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:33.750704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 485/977] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:33 2022] GET /identity/v3/auth/tokens => generated 1570 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 29 02:00:33.756365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:33.823225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:33.863686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bba064-e1d6-4b4c-b118-2e021f0a3563 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:33.899312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:33.900272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:33.900738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:33.901219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:33.912466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:33.914359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:33.930426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:33.931462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:33.932372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:33.932372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:33.932372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:33.938684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 493/978] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:33 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:33.940010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:33.994870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:34.043841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e967a01c-3b44-493a-879d-dbce4fc22367 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:34.078470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:34.079087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.079362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.079630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.080275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:34.080575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:34.081952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:34.093553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 486/979] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:34.099353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:34.200986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd340048-df03-4f19-9806-6e38e65b42ed None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:34.217072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:34.237007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:34.237657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.237935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.238241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.245810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:34.247290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:34.258718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:34.259694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:34.260595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:34.264662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 494/980] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:34 2022] GET /identity/v3/projects/service => generated 88 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:34.266149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:34.369602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896b08e0-5879-4f46-b19a-492b7177ebe4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:34.384007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:34.407670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed80d64-e94b-48b2-a0ec-85d9f340615a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:34.408331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed80d64-e94b-48b2-a0ec-85d9f340615a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.408582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed80d64-e94b-48b2-a0ec-85d9f340615a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.408879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed80d64-e94b-48b2-a0ec-85d9f340615a None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.409738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed80d64-e94b-48b2-a0ec-85d9f340615a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:34.410031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed80d64-e94b-48b2-a0ec-85d9f340615a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:34.411669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed80d64-e94b-48b2-a0ec-85d9f340615a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:34.430070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 487/981] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:34 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:34.431688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:34.540970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5637bd8b-2d24-4577-b1bc-6d74567f7656 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:34.580063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a443675-4509-4106-9a43-95d9820f7014 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:34.581024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a443675-4509-4106-9a43-95d9820f7014 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.581514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a443675-4509-4106-9a43-95d9820f7014 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.581962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a443675-4509-4106-9a43-95d9820f7014 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.583007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a443675-4509-4106-9a43-95d9820f7014 None admin] RBAC: Adding query filter params (role.id=3495b485e650453790bdb49f328d4305, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=d505d9008c9540659a3de626874c2457) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:34.583505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a443675-4509-4106-9a43-95d9820f7014 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:34.585217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a443675-4509-4106-9a43-95d9820f7014 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:34.591765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 495/982] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:34 2022] GET /identity/v3/role_assignments?user.id=d505d9008c9540659a3de626874c2457&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=3495b485e650453790bdb49f328d4305 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:34.950139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17fb77a-12f1-4688-acf6-cdf9310114bb None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.950669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17fb77a-12f1-4688-acf6-cdf9310114bb None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.950669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17fb77a-12f1-4688-acf6-cdf9310114bb None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.952062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 488/983] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:34.957406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab80a9-a29d-4aee-a33b-0f375fd864f5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:34.957708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab80a9-a29d-4aee-a33b-0f375fd864f5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:34.957983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28ab80a9-a29d-4aee-a33b-0f375fd864f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:34.976770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-28ab80a9-a29d-4aee-a33b-0f375fd864f5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:34.977962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-28ab80a9-a29d-4aee-a33b-0f375fd864f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:35.015894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:35.051787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 496/984] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:34 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:35.314396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17fb77a-12f1-4688-acf6-cdf9310114bb None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.424861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17fb77a-12f1-4688-acf6-cdf9310114bb None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.460027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9952bde-9c9a-42d8-ac65-1c7768ecf65e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.460654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9952bde-9c9a-42d8-ac65-1c7768ecf65e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.460921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9952bde-9c9a-42d8-ac65-1c7768ecf65e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.461184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9952bde-9c9a-42d8-ac65-1c7768ecf65e None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.461755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9952bde-9c9a-42d8-ac65-1c7768ecf65e None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.463156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9952bde-9c9a-42d8-ac65-1c7768ecf65e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-e9952bde-9c9a-42d8-ac65-1c7768ecf65e None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:35.471748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:35.472710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:35.473710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:35.477979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 489/985] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:35.479397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab80a9-a29d-4aee-a33b-0f375fd864f5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.519728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c822c64e-ef76-4dd1-8979-e59636875760 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.519980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c822c64e-ef76-4dd1-8979-e59636875760 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.520335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c822c64e-ef76-4dd1-8979-e59636875760 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.521699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 490/986] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:35.527180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57943674-cae1-4667-9feb-e408b39bc81e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.527516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57943674-cae1-4667-9feb-e408b39bc81e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.527777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57943674-cae1-4667-9feb-e408b39bc81e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.550038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-57943674-cae1-4667-9feb-e408b39bc81e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:35.551264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-57943674-cae1-4667-9feb-e408b39bc81e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:35.581532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:35.597023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28ab80a9-a29d-4aee-a33b-0f375fd864f5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.626029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 491/987] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:35 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:35.635744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.636424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.636767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.637107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.637824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:35.638240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.639759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.644892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 497/988] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:35.650604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57943674-cae1-4667-9feb-e408b39bc81e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.750766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57943674-cae1-4667-9feb-e408b39bc81e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.787265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.787876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.788147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.788411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.789034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:35.789334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.790732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.795875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 492/989] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:35.802035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.837794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:35.889057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:35.908660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbe6c01-38be-40b8-9216-93a702030ca3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:35.923258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:35.946033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:35.946783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:35.947048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:35.947611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:35.948311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] RBAC: Adding query filter params (interface=admin, service_id=bb0537d91e6a42f5a3f7ec33c184f688, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:35.948615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:35.950495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:35.956473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 498/990] 10.222.0.61 () {56 vars in 1249 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/endpoints?service_id=bb0537d91e6a42f5a3f7ec33c184f688&interface=admin®ion_id=RegionOne => generated 193 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:35.962459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:36.002929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dce5295-55bd-4a01-9531-381789394a41 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:36.038927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:36.039563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.039839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.040107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.040743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:36.042597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:36.053185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:36.054225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:36.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:36.059422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 493/991] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/services/compute => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:36.060272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:36.065078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:36.069952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1c6f918-cb62-47fa-8d52-a4a3eee86458 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:36.113924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:36.115485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.115767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.116288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.116899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:36.118505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:36.128066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 499/992] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:35 2022] GET /identity/v3/services => generated 1727 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:00:36.169204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:36.178986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-846cbf64-f577-4a35-8ebc-d296260b56e1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:36.217559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:36.218242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.218519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.218793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.219435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] RBAC: Adding query filter params (name=compute) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:36.219736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:36.221146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:36.227301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 494/993] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:36 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:36.233087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:36.342124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5362f9-5691-42b4-8fa5-16ccd7ba93b7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:36.377471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:36.378025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.378302 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.378565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.379187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:36.379571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:36.381039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:36.385844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 500/994] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:36 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:36.392070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:36.498430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ba3d483-f069-4140-80ee-f6bc95be622e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:36.538964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0880ea17-2b55-42e7-bb5f-879fb61be086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:36.539588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0880ea17-2b55-42e7-bb5f-879fb61be086 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.539865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0880ea17-2b55-42e7-bb5f-879fb61be086 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.540189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0880ea17-2b55-42e7-bb5f-879fb61be086 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.540897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0880ea17-2b55-42e7-bb5f-879fb61be086 None admin] RBAC: Adding query filter params (interface=public, service_id=7dd33775d85a415f9b4fd66272ed3553, region_id=RegionOne) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:36.541203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0880ea17-2b55-42e7-bb5f-879fb61be086 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:36.545293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0880ea17-2b55-42e7-bb5f-879fb61be086 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:36.552013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 495/995] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:00:36 2022] GET /identity/v3/endpoints?service_id=7dd33775d85a415f9b4fd66272ed3553&interface=public®ion_id=RegionOne => generated 194 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:36.558702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:36.587207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:36.683373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f975eafb-4c6f-4576-82bd-4014bdffcf45 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:36.722054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5ed9af-cd88-409f-bad3-7958b16c4490 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:36.722647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ed9af-cd88-409f-bad3-7958b16c4490 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.722924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ed9af-cd88-409f-bad3-7958b16c4490 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.723286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5ed9af-cd88-409f-bad3-7958b16c4490 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.723895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5ed9af-cd88-409f-bad3-7958b16c4490 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:36.725265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5ed9af-cd88-409f-bad3-7958b16c4490 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:36.731764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 501/996] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:36 2022] GET /identity/v3/services => generated 1727 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:00:36.797675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89f26a-06bb-4846-8568-be462ad1bb9d None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.798075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89f26a-06bb-4846-8568-be462ad1bb9d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.798237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d89f26a-06bb-4846-8568-be462ad1bb9d None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.799778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 496/997] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:36.805926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72994e4f-84d6-47c4-8f00-19fb89a22c5e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:36.806206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72994e4f-84d6-47c4-8f00-19fb89a22c5e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:36.806506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72994e4f-84d6-47c4-8f00-19fb89a22c5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:36.826500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-72994e4f-84d6-47c4-8f00-19fb89a22c5e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:36.827734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-72994e4f-84d6-47c4-8f00-19fb89a22c5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:36.904466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 502/998] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:36 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:37.165644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d89f26a-06bb-4846-8568-be462ad1bb9d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:37.193436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:37.277885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d89f26a-06bb-4846-8568-be462ad1bb9d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:37.311990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:37.312622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:37.312885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:37.313141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] PATH_INFO: `/v3/services/image` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:37.313733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:37.315139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:37.324461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:37.325441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:37.326810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:37.330074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 497/999] 10.222.0.61 () {56 vars in 1105 bytes} [Thu Sep 29 02:00:37 2022] GET /identity/v3/services/image => generated 86 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:37.331880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72994e4f-84d6-47c4-8f00-19fb89a22c5e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:37.353696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:37.436759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72994e4f-84d6-47c4-8f00-19fb89a22c5e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:37.471106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fea82cd-143d-4196-bc2f-1cc80313cad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:37.471713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fea82cd-143d-4196-bc2f-1cc80313cad1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:37.471983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fea82cd-143d-4196-bc2f-1cc80313cad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:37.472249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fea82cd-143d-4196-bc2f-1cc80313cad1 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:37.472878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fea82cd-143d-4196-bc2f-1cc80313cad1 None admin] RBAC: Adding query filter params (name=image) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:37.473261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fea82cd-143d-4196-bc2f-1cc80313cad1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:37.474744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fea82cd-143d-4196-bc2f-1cc80313cad1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:37.484305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 503/1000] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:37 2022] GET /identity/v3/services?name=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:37.485480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:37.577537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:37.593134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2c4b630-e0d7-4e5d-9491-b37267dd93fe None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:37.629557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85f0424c-8d78-41d1-92ff-1018ea2ea1c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:37.630366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0424c-8d78-41d1-92ff-1018ea2ea1c0 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:37.630638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0424c-8d78-41d1-92ff-1018ea2ea1c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:37.630905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85f0424c-8d78-41d1-92ff-1018ea2ea1c0 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:37.631527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f0424c-8d78-41d1-92ff-1018ea2ea1c0 None admin] RBAC: Adding query filter params (type=image) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:37.631824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f0424c-8d78-41d1-92ff-1018ea2ea1c0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:37.633198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85f0424c-8d78-41d1-92ff-1018ea2ea1c0 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:37.637974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 498/1001] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:37 2022] GET /identity/v3/services?type=image => generated 122 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:38.337930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b25f3-9a95-425e-b7a4-79c4845e2abe None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.338487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b25f3-9a95-425e-b7a4-79c4845e2abe None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.338487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc4b25f3-9a95-425e-b7a4-79c4845e2abe None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.339901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 504/1002] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:38.345421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a65ab-5f76-46dd-98de-8201a10c23b3 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.345982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a65ab-5f76-46dd-98de-8201a10c23b3 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.345982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424a65ab-5f76-46dd-98de-8201a10c23b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.364904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-424a65ab-5f76-46dd-98de-8201a10c23b3 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:38.366125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-424a65ab-5f76-46dd-98de-8201a10c23b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:38.438066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 499/1003] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:38 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:38.705142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4b25f3-9a95-425e-b7a4-79c4845e2abe None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.812020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:38.830723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc4b25f3-9a95-425e-b7a4-79c4845e2abe None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:38.865630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:38.866285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.866556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.866821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.867398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:38.868807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:38.877815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:38.878897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:38.883177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 505/1004] 10.222.0.61 () {56 vars in 1126 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:38.884064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e4aaad-bd53-4fd7-a3f6-a2ab10ac08af None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.884064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e4aaad-bd53-4fd7-a3f6-a2ab10ac08af None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.884064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e4aaad-bd53-4fd7-a3f6-a2ab10ac08af None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.884064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 500/1005] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:38.884746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:38.889626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5bb87e-7e00-4c94-af62-d82565d87d7c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:38.889973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5bb87e-7e00-4c94-af62-d82565d87d7c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:38.890417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e5bb87e-7e00-4c94-af62-d82565d87d7c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:38.909070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7e5bb87e-7e00-4c94-af62-d82565d87d7c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:38.910308 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7e5bb87e-7e00-4c94-af62-d82565d87d7c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:38.978758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 501/1006] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:38 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:38.994025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-581a2c04-7b66-4ec1-980d-d64b168cbe07 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.030886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.031899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.032363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.032842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.033943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:39.034312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.035817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.042314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 506/1007] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:39.048008 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5bb87e-7e00-4c94-af62-d82565d87d7c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:39.060929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:39.153903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e5bb87e-7e00-4c94-af62-d82565d87d7c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.189354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.189973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.190270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.190542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.191168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:39.191468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.193032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.198077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 502/1008] 10.222.0.61 () {56 vars in 1122 bytes} [Thu Sep 29 02:00:39 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:39.203738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:39.219362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:39.268160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:39.316906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e845359-de07-4cc9-b3bd-0348ad54cb21 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.352600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:39.353951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.354592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.354871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.355527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.356529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.358053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.379133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 507/1009] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:39 2022] POST /identity/v3/endpoints => generated 325 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:39.380355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb24749-8036-422d-938a-cecba39dda71 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.436522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.437166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.437439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.437690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] PATH_INFO: `/v3/services/compute` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.438304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.439721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:39.449737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:39.450929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:39.455196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 503/1010] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:39 2022] GET /identity/v3/services/compute => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:39.457801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:39.546133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:39.583083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d8d003d-b606-4f9d-92fb-fd303679f1a2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.622316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0553d5f7-14fc-4fb5-973c-a6ae3be8c9d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.622933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0553d5f7-14fc-4fb5-973c-a6ae3be8c9d1 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.623208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0553d5f7-14fc-4fb5-973c-a6ae3be8c9d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.623479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0553d5f7-14fc-4fb5-973c-a6ae3be8c9d1 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.624110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0553d5f7-14fc-4fb5-973c-a6ae3be8c9d1 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:39.624412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0553d5f7-14fc-4fb5-973c-a6ae3be8c9d1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.625799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0553d5f7-14fc-4fb5-973c-a6ae3be8c9d1 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.632090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 508/1011] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:39 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:39.637519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:39.743599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc96699-6d8d-40f4-937e-9c87415cb44e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.759538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a0438d-2509-4f66-9d54-885690107832 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.759822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a0438d-2509-4f66-9d54-885690107832 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.760096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a0438d-2509-4f66-9d54-885690107832 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.761460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 509/1012] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:39.766471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39916ea6-b320-4247-87d9-b188dc5d08c6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.766746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39916ea6-b320-4247-87d9-b188dc5d08c6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.767002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39916ea6-b320-4247-87d9-b188dc5d08c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.780091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.780714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.780989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.781262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.781892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] RBAC: Adding query filter params (type=compute) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:39.782220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.783603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.784971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:39.787252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-39916ea6-b320-4247-87d9-b188dc5d08c6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:39.788603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-39916ea6-b320-4247-87d9-b188dc5d08c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:39.794875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 504/1013] 10.222.0.61 () {56 vars in 1112 bytes} [Thu Sep 29 02:00:39 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:39.796409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:39.874240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 510/1014] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:39 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:39.902549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18279c3c-5177-44ab-9a05-6fba0f8d259c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:39.938078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-983d2ebb-abdf-4898-af47-fcd22c87969a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:39.938684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d2ebb-abdf-4898-af47-fcd22c87969a None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:39.938961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d2ebb-abdf-4898-af47-fcd22c87969a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:39.939229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-983d2ebb-abdf-4898-af47-fcd22c87969a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:39.939883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d2ebb-abdf-4898-af47-fcd22c87969a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:39.941376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-983d2ebb-abdf-4898-af47-fcd22c87969a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:39.976801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 505/1015] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:39 2022] POST /identity/v3/endpoints => generated 334 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:40.128396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39916ea6-b320-4247-87d9-b188dc5d08c6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:40.200782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:40.238081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39916ea6-b320-4247-87d9-b188dc5d08c6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:40.279886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285c9617-d008-4b03-990d-8236a56b7f2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:40.280502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c9617-d008-4b03-990d-8236a56b7f2c None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:40.280772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c9617-d008-4b03-990d-8236a56b7f2c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:40.281051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285c9617-d008-4b03-990d-8236a56b7f2c None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:40.281686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285c9617-d008-4b03-990d-8236a56b7f2c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:40.283113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285c9617-d008-4b03-990d-8236a56b7f2c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:40.301989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 511/1016] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:40 2022] POST /identity/v3/services => generated 242 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:41.548559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b13545-7eef-4131-aee4-1b28027f1553 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:41.549012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b13545-7eef-4131-aee4-1b28027f1553 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:41.549094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b13545-7eef-4131-aee4-1b28027f1553 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:41.550623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 506/1017] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:41.556376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082cb5b-8df7-4ef1-ae04-2edba1f4a3fc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:41.556850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082cb5b-8df7-4ef1-ae04-2edba1f4a3fc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:41.556945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9082cb5b-8df7-4ef1-ae04-2edba1f4a3fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:41.576234 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9082cb5b-8df7-4ef1-ae04-2edba1f4a3fc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:41.577407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9082cb5b-8df7-4ef1-ae04-2edba1f4a3fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:41.654044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 512/1018] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:41 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:41.918872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b13545-7eef-4131-aee4-1b28027f1553 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:41.977473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:42.024355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b13545-7eef-4131-aee4-1b28027f1553 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:42.058242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fdbafae-e04f-42be-a3b5-713e0c5d4b3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:42.058865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdbafae-e04f-42be-a3b5-713e0c5d4b3b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.059135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdbafae-e04f-42be-a3b5-713e0c5d4b3b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.059401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fdbafae-e04f-42be-a3b5-713e0c5d4b3b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.060506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdbafae-e04f-42be-a3b5-713e0c5d4b3b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:42.061970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fdbafae-e04f-42be-a3b5-713e0c5d4b3b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:42.063714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 507/1019] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:41 2022] GET /identity/v3/domains/default => generated 199 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:42.069617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9082cb5b-8df7-4ef1-ae04-2edba1f4a3fc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:42.118080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:42.172880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9082cb5b-8df7-4ef1-ae04-2edba1f4a3fc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:42.210717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b7b4d1-1a42-4b70-b790-b11db9cd520b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:42.211330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b7b4d1-1a42-4b70-b790-b11db9cd520b None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.211597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b7b4d1-1a42-4b70-b790-b11db9cd520b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.211858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b7b4d1-1a42-4b70-b790-b11db9cd520b None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.212512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b7b4d1-1a42-4b70-b790-b11db9cd520b None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:42.213950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b7b4d1-1a42-4b70-b790-b11db9cd520b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:42.250034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 513/1020] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:00:42 2022] POST /identity/v3/projects => generated 311 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:42.296009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff275d8-c40b-4bac-baa7-b31f02ad2f25 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.296296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff275d8-c40b-4bac-baa7-b31f02ad2f25 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.296551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff275d8-c40b-4bac-baa7-b31f02ad2f25 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.298104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 508/1021] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:42.305072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db59752-0619-4949-bb01-38e9030f2404 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.305362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db59752-0619-4949-bb01-38e9030f2404 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.305623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db59752-0619-4949-bb01-38e9030f2404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.326094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9db59752-0619-4949-bb01-38e9030f2404 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:42.327320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9db59752-0619-4949-bb01-38e9030f2404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:42.415911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 514/1022] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:42 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:42.469110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c54f4d-b49b-46ea-a6f5-b0c72e285f2c None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.469295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c54f4d-b49b-46ea-a6f5-b0c72e285f2c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.469571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1c54f4d-b49b-46ea-a6f5-b0c72e285f2c None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.470965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 509/1023] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:42.476577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f5fe36-45e5-4f54-a7b3-d0429684d804 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.476826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f5fe36-45e5-4f54-a7b3-d0429684d804 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.477111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0f5fe36-45e5-4f54-a7b3-d0429684d804 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.496778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d0f5fe36-45e5-4f54-a7b3-d0429684d804 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:42.497984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d0f5fe36-45e5-4f54-a7b3-d0429684d804 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:42.525174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:42.582035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 515/1024] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:42 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:42.689378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c54f4d-b49b-46ea-a6f5-b0c72e285f2c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:42.796210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1c54f4d-b49b-46ea-a6f5-b0c72e285f2c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:42.820021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:42.832701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:42.833330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.833611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.833876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.844298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f5fe36-45e5-4f54-a7b3-d0429684d804 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:42.845565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:42.850534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:42.851501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:42.852340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 510/1025] 10.222.0.61 () {56 vars in 1120 bytes} [Thu Sep 29 02:00:42 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:42.858021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:42.958411 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0f5fe36-45e5-4f54-a7b3-d0429684d804 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:42.963099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7998c830-6b74-4ab8-8c07-9f1ebd592df4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:42.994523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:42.995167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.995441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.995710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] PATH_INFO: `/v3/services/image` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.996362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:42.997662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:42.997916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:42.998560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:42.998840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:42.999110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:42.999872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:43.000276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.001706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:43.006666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:43.007705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:43.008553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:43.012767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 516/1026] 10.222.0.61 () {56 vars in 1105 bytes} [Thu Sep 29 02:00:42 2022] GET /identity/v3/services/image => generated 86 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:43.013593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.018041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 511/1027] 10.222.0.61 () {56 vars in 1115 bytes} [Thu Sep 29 02:00:42 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:43.021376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.043370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:43.070784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:43.136169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8817294-84b3-4393-9756-528de605cf54 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.136386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93857499-caac-4be7-a5e7-672dee42582b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.174079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.174687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.174793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.175011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.175220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.175414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.175744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.175838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] RBAC: Adding query filter params (name=image) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:43.176189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.176730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.177983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.188041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 517/1028] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/services?name=image => generated 122 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:43.189148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.193336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.194932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:43.211275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:43.212348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:43.213231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:43.213231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:43.213231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:43.213231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 512/1029] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:43.218433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.286172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:43.299376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35e204ad-c94c-4aea-a4c2-436e89d65b0d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.325890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:43.332857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08cf1925-486c-44c5-9c6c-a436586d689b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.338594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.339427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.340265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.340710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.341514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] RBAC: Adding query filter params (type=image) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:43.342336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.344434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.350711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 518/1030] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/services?type=image => generated 350 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:43.356309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.368454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.369070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.369344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.369613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.370282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:43.370611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.371941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.385226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 513/1031] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:43.390787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.463898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d927b0d-5600-47e9-b8af-7469406acacc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.494118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdb381c9-b6c9-4ae3-8294-4b70604ceaca None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.500437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.501049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.501323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.501599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.502347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] RBAC: Adding query filter params (interface=public, service_id=7039527209da42a791957ba216b2f4cb, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:43.502653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.504048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.514558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 519/1032] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/endpoints?service_id=7039527209da42a791957ba216b2f4cb&interface=public®ion_id=RegionOne => generated 194 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:43.515361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.536613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:43.538987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.539662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.539969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.540223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] PATH_INFO: `/v3/users/nova` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.552076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.554689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:43.569446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:43.570723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:43.571668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:43.575679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 514/1033] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:43.576450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.595102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:43.630500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93f9f5d-ea76-4f20-acc0-e228b7b94b4e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.667296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.667908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.668185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.668453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.669054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.670463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.678379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8acb8b20-8bf6-4dca-b3cf-8b396a8cf26a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.686536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 520/1034] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/services => generated 1957 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:00:43.715502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.716229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.716398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.716651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.717381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:43.717680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.719207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:43.740815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 515/1035] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:43.746721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:43.835750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:43.859556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1a0fb3e-5b81-40b9-88cb-c381c8a05fd5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:43.893738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:43.894420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:43.894733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:43.894998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:43.964701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:43.966174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:44.047615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:44.070087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 521/1036] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:43 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:44.075434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:44.153088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:44.181638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-378e507c-006b-450d-bd76-96485f0c4321 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:44.215320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:44.215946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:44.216215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:44.216476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:44.227017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:44.228465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:44.243249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:44.244279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:44.245176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:44.245176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:44.245176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:44.250043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 516/1037] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:44.250177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:44.351092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26ee9b5e-e242-4e95-b981-2946c2a4ad15 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:44.389496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a646453-8d42-4cac-a51e-66279681044f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:44.390130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a646453-8d42-4cac-a51e-66279681044f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:44.390402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a646453-8d42-4cac-a51e-66279681044f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:44.390670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a646453-8d42-4cac-a51e-66279681044f None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:44.391319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a646453-8d42-4cac-a51e-66279681044f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:44.391616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a646453-8d42-4cac-a51e-66279681044f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:44.392980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a646453-8d42-4cac-a51e-66279681044f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:44.405032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 522/1038] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:44.410897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:44.472732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d121f3-25b6-4b54-b960-32654a71bf5b None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:44.473118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d121f3-25b6-4b54-b960-32654a71bf5b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:44.473622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44d121f3-25b6-4b54-b960-32654a71bf5b None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:44.475145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 523/1039] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:44.480335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818edb8-2da5-49f4-a58c-b545e635c4e0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:44.480608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818edb8-2da5-49f4-a58c-b545e635c4e0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:44.480869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6818edb8-2da5-49f4-a58c-b545e635c4e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:44.499743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6818edb8-2da5-49f4-a58c-b545e635c4e0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:44.500928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6818edb8-2da5-49f4-a58c-b545e635c4e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:44.521905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3d528d0-f8dd-4166-9ef3-4b5fcef3d7f7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:44.540669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:44.556611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:44.566566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:44.567451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:44.567561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:44.567762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:44.575355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:44.576823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:44.586308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 524/1040] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:44 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:44.588758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:44.589687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:44.590634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:44.594294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 517/1041] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:44 2022] GET /identity/v3/projects/service => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:44.595707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6818edb8-2da5-49f4-a58c-b545e635c4e0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:44.693721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6818edb8-2da5-49f4-a58c-b545e635c4e0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:44.727500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:44.728101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:44.728370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:44.728637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:44.729321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:44.729622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:44.731085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:44.742774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 525/1042] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:44 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:44.748839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:44.845172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:44.847359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facce0c9-3b58-473c-aeae-a7a5f207aec5 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:44.875880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:44.881580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:44.882227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:44.882505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:44.882776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:44.883701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] RBAC: Adding query filter params (role.id=3495b485e650453790bdb49f328d4305, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=bc807d5f532b4e4c9953ba5e7eb878bb) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:44.884005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:44.885457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:44.892755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 518/1043] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:44 2022] GET /identity/v3/role_assignments?user.id=bc807d5f532b4e4c9953ba5e7eb878bb&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=3495b485e650453790bdb49f328d4305 => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:44.966400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55839a35-bd30-42fb-a7fe-7288743b9d35 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:45.005132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1897be79-d9a4-40b3-8a92-688856517093 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:45.005848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897be79-d9a4-40b3-8a92-688856517093 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:45.006157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897be79-d9a4-40b3-8a92-688856517093 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:45.006430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897be79-d9a4-40b3-8a92-688856517093 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:45.007598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1897be79-d9a4-40b3-8a92-688856517093 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:45.009030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1897be79-d9a4-40b3-8a92-688856517093 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:45.014220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 526/1044] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:44 2022] GET /identity/v3/domains/default => generated 199 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:45.017032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:45.073677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:45.126393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbd3bd0d-0124-4f36-b5a6-a5e25818b17a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:45.160990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df816280-f969-4ee1-ab47-035435306702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:45.161606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df816280-f969-4ee1-ab47-035435306702 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:45.161883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df816280-f969-4ee1-ab47-035435306702 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:45.162183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df816280-f969-4ee1-ab47-035435306702 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:45.162836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df816280-f969-4ee1-ab47-035435306702 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:45.164282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df816280-f969-4ee1-ab47-035435306702 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:45.189932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 519/1045] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 02:00:45 2022] POST /identity/v3/users => generated 285 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:45.836271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b830b157-eae5-44b9-a063-9f2a2440cba6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:45.836798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b830b157-eae5-44b9-a063-9f2a2440cba6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:45.836798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b830b157-eae5-44b9-a063-9f2a2440cba6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:45.838348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 527/1046] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:45.843587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdddc7-bace-485d-97e9-5b6b54a7ad48 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:45.844005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdddc7-bace-485d-97e9-5b6b54a7ad48 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:45.844082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79fdddc7-bace-485d-97e9-5b6b54a7ad48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:45.863833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-79fdddc7-bace-485d-97e9-5b6b54a7ad48 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:45.865100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-79fdddc7-bace-485d-97e9-5b6b54a7ad48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:45.933515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 520/1047] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:45 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:46.191758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b830b157-eae5-44b9-a063-9f2a2440cba6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:46.290119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:46.296009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b830b157-eae5-44b9-a063-9f2a2440cba6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:46.329530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:46.330181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:46.330449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:46.330719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] PATH_INFO: `/v3/services/image` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:46.331292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:46.332749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:46.341534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:46.342554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 29 02:00:46.343409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:46.346366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 528/1048] 10.222.0.61 () {56 vars in 1105 bytes} [Thu Sep 29 02:00:46 2022] GET /identity/v3/services/image => generated 86 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:46.348683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fdddc7-bace-485d-97e9-5b6b54a7ad48 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:46.382828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:46.450333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79fdddc7-bace-485d-97e9-5b6b54a7ad48 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:46.484917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:46.485523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:46.485793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:46.486184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:46.486716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] RBAC: Adding query filter params (name=image) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:46.487006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:46.488430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:46.498384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 521/1049] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:46 2022] GET /identity/v3/services?name=image => generated 122 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:46.499253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:46.611247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7388cf71-0545-47ec-be15-5ddb03c33c25 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:46.645835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33124107-2eb1-4c50-bf19-90014b37cde2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:46.646747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33124107-2eb1-4c50-bf19-90014b37cde2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:46.647232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33124107-2eb1-4c50-bf19-90014b37cde2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:46.647666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33124107-2eb1-4c50-bf19-90014b37cde2 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:46.648454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33124107-2eb1-4c50-bf19-90014b37cde2 None admin] RBAC: Adding query filter params (type=image) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:46.648921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33124107-2eb1-4c50-bf19-90014b37cde2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:46.650484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33124107-2eb1-4c50-bf19-90014b37cde2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:46.655707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 529/1050] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:46 2022] GET /identity/v3/services?type=image => generated 350 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:46.662157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:46.762760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:46.766685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acaeb2ba-60d1-405c-8e79-325f25b6fb70 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:46.803194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a131c7fe-fd80-4f43-a446-e0247bfe6244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:46.803802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a131c7fe-fd80-4f43-a446-e0247bfe6244 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:46.804078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a131c7fe-fd80-4f43-a446-e0247bfe6244 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:46.804345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a131c7fe-fd80-4f43-a446-e0247bfe6244 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:46.805015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a131c7fe-fd80-4f43-a446-e0247bfe6244 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:46.806441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a131c7fe-fd80-4f43-a446-e0247bfe6244 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:46.834052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 522/1051] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:00:46 2022] POST /identity/v3/endpoints => generated 327 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:00:47.038660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d47779e-0004-4d7a-928f-780aaf5a9370 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.039167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d47779e-0004-4d7a-928f-780aaf5a9370 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.039167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d47779e-0004-4d7a-928f-780aaf5a9370 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.040449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 530/1052] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:47.046292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301dd8b1-e393-4e3b-9cd3-f38eb9700e04 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.046748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301dd8b1-e393-4e3b-9cd3-f38eb9700e04 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.046835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301dd8b1-e393-4e3b-9cd3-f38eb9700e04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.065830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-301dd8b1-e393-4e3b-9cd3-f38eb9700e04 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:47.067275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-301dd8b1-e393-4e3b-9cd3-f38eb9700e04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:47.141123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 523/1053] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:47.408750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d47779e-0004-4d7a-928f-780aaf5a9370 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:47.427798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355eaa3a-ee5c-48d1-b48b-d6654a94dd3c None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.428052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355eaa3a-ee5c-48d1-b48b-d6654a94dd3c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.428321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355eaa3a-ee5c-48d1-b48b-d6654a94dd3c None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.429813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 524/1054] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:47.434626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3d9243-8b88-459e-90f2-44154d4ea3e4 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.434922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3d9243-8b88-459e-90f2-44154d4ea3e4 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.435178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3d9243-8b88-459e-90f2-44154d4ea3e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.440173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:47.454640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-df3d9243-8b88-459e-90f2-44154d4ea3e4 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:47.456082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-df3d9243-8b88-459e-90f2-44154d4ea3e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:47.516193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d47779e-0004-4d7a-928f-780aaf5a9370 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:47.528107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 525/1055] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:47.558222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:47.558945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.559288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.559623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.571241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:47.576095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:47.577041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:47.577878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 531/1056] 10.222.0.61 () {56 vars in 1120 bytes} [Thu Sep 29 02:00:47 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:47.583630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3d9243-8b88-459e-90f2-44154d4ea3e4 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:47.599168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:47.688755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3d9243-8b88-459e-90f2-44154d4ea3e4 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:47.722093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:47.722731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.722998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.723261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.723914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:47.724205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:47.725571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:47.733782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 526/1057] 10.222.0.61 () {56 vars in 1115 bytes} [Thu Sep 29 02:00:47 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:47.739661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:47.810024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:47.828076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:47.844120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cda2457-d291-4a3a-8b0e-398e1a466d8d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:47.879565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:47.880184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.880467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.880713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.892985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:47.894768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:47.897302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:47.911725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:47.912722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:47.914509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:47.914509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:47.914509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:47.914509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 532/1058] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:47.918549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:47.928429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c486e8b-e5d6-47d1-8940-82f38d466d86 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:47.964447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:47.965093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:47.965374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:47.965640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:47.976201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:47.981501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:47.982471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:47.983302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 527/1059] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:47.988740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.030750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcbc60d-d619-4d55-a1e0-7542a3fa3dd2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.068647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.069276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.069528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.069767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.070436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:48.070735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.072098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.080159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.086275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 533/1060] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:48.091771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.092652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dc8b0c0-ac12-42fa-8c2a-2f9c1db4dc78 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.130347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.131245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.131531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.131802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.132462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:48.132905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.134966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.137147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.145582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 528/1061] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:48.151312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.203654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d008cd9-202c-448b-b8d0-c02bfbe4cf7d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.239580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.240125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.240394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.240928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] PATH_INFO: `/v3/users/nova` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.247779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.249201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.253013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4a2a96-06ee-4ad6-a7e4-60e120550d0f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:48.261083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:48.262072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:48.263027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:48.266310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 534/1062] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/users/nova => generated 82 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:48.267839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.290017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.290652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.290921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.291183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] PATH_INFO: `/v3/users/c8c8d217f6cc4475997ba7a82434df14` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.305847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] RBAC: Authorizing `identity:get_user(user_id=c8c8d217f6cc4475997ba7a82434df14)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.307308 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.313874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 529/1063] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/users/c8c8d217f6cc4475997ba7a82434df14 => generated 285 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:48.314512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.335309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.373313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.382211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14e452a7-6789-40a1-90b5-20a6a673d81f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.418184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.418795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.419069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.419336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.420061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:48.420360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.421413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83fd64d-4edd-4c39-bd3e-024bb86ab2e2 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.421783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.440626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 535/1064] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:48.447018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.458196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.459069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.459636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.459980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.534977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.536401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.557292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b48285b-2041-4132-bf9c-b76d2feafc46 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.575959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.595072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.595267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.595923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.596306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.596538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.607894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.609430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:48.627025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:48.628002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:48.629000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:48.629000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:48.629000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:48.634719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 536/1065] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:48.635628 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.656683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 530/1066] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 29 02:00:48.662380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.748317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d6a298-605f-47ed-b71a-5587d79a7ad6 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.765679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64078b37-0cf2-4c16-887b-3ce5326ce4fe None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.785611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.785809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.786410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.786410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.786925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:48.787210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.788579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.800283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.806301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 537/1067] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:48.806438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.806438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.806438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.807411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] RBAC: Authorizing `identity:get_project(project_id=2a1bc981b8d445fe91329ba5f3e99775)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.807527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.808840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.810649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 531/1068] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:48.817831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:48.840461 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.844299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:48.922086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb0f2d6-92c0-4206-8d92-1723b03a5a61 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.932123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfd8dadd-628c-4f23-8ad8-085114ddee4b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:48.959227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29590558-f299-4080-b2e5-b27e51e3d53b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.959852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29590558-f299-4080-b2e5-b27e51e3d53b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.960124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29590558-f299-4080-b2e5-b27e51e3d53b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.960387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29590558-f299-4080-b2e5-b27e51e3d53b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.967178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29590558-f299-4080-b2e5-b27e51e3d53b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.968595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29590558-f299-4080-b2e5-b27e51e3d53b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.971632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:48.972236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:48.972556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:48.972824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:48.973539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=2a1bc981b8d445fe91329ba5f3e99775, user.id=c8c8d217f6cc4475997ba7a82434df14) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:48.973843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:48.975301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-29590558-f299-4080-b2e5-b27e51e3d53b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:48.980142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:48.988269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:48.989361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:48.989795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 532/1069] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/role_assignments?user.id=c8c8d217f6cc4475997ba7a82434df14&scope.project.id=2a1bc981b8d445fe91329ba5f3e99775&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:00:48.989871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 538/1070] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:48.989952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:49.027170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ddf5d5-d935-4309-bdbf-6cd237434a94 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:49.027473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ddf5d5-d935-4309-bdbf-6cd237434a94 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:49.027744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ddf5d5-d935-4309-bdbf-6cd237434a94 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:49.030043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 539/1071] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:49.037001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0f7e8-5715-4168-90ce-0e4e0f32830f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:49.037183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0f7e8-5715-4168-90ce-0e4e0f32830f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:49.037442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e0f7e8-5715-4168-90ce-0e4e0f32830f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:49.059767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e5e0f7e8-5715-4168-90ce-0e4e0f32830f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:49.061000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e5e0f7e8-5715-4168-90ce-0e4e0f32830f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:49.091368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:49.110212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31e77485-e34e-41ad-bd42-411d64199126 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:49.140336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 540/1072] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:49.156311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:49.156943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:49.157219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:49.157487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:49.158248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:49.158603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:49.160411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:49.174826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 533/1073] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:49.181124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e0f7e8-5715-4168-90ce-0e4e0f32830f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:49.200998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:49.287248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e0f7e8-5715-4168-90ce-0e4e0f32830f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:49.322563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:49.323210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:49.323453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:49.323733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/bc807d5f532b4e4c9953ba5e7eb878bb/roles/3495b485e650453790bdb49f328d4305` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:49.325992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=bc807d5f532b4e4c9953ba5e7eb878bb, role_id=3495b485e650453790bdb49f328d4305)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:49.327568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:49.344105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 541/1074] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:49 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/bc807d5f532b4e4c9953ba5e7eb878bb/roles/3495b485e650453790bdb49f328d4305 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:00:49.412724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:49.534029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb56afd-a57b-476a-8b21-b267186b3481 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:49.572197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:49.572817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:49.573094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:49.573365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:49.588908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:49.644096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:49.645615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:49.754059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 534/1075] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 339 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:49.757483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:49.841591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:49.866326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9899270b-f1a6-42be-896e-483a8750572c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:49.904496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d27b05-9804-4ecd-a8d8-33fa04258c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:49.905123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d27b05-9804-4ecd-a8d8-33fa04258c23 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:49.905392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d27b05-9804-4ecd-a8d8-33fa04258c23 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:49.905657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d27b05-9804-4ecd-a8d8-33fa04258c23 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:49.916216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d27b05-9804-4ecd-a8d8-33fa04258c23 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:49.917653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d27b05-9804-4ecd-a8d8-33fa04258c23 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-97d27b05-9804-4ecd-a8d8-33fa04258c23 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:49.933058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:49.934063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:49.934989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:49.934989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:49.934989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:49.935710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 542/1076] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:49.940198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:49.990433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:50.044570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c635fb66-2c62-4036-b923-5441d7fb1be4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:50.078738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab5b924b-6100-46cb-b317-ef1fe6852156 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:50.079342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5b924b-6100-46cb-b317-ef1fe6852156 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:50.079609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5b924b-6100-46cb-b317-ef1fe6852156 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:50.079872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab5b924b-6100-46cb-b317-ef1fe6852156 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:50.080502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5b924b-6100-46cb-b317-ef1fe6852156 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:50.080794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5b924b-6100-46cb-b317-ef1fe6852156 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:50.082232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab5b924b-6100-46cb-b317-ef1fe6852156 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:50.093714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 535/1077] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:51.127580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3cc87-9c49-480b-b1ca-e207253613f7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:51.128205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3cc87-9c49-480b-b1ca-e207253613f7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:51.128205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c3cc87-9c49-480b-b1ca-e207253613f7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:51.129445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 543/1078] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:51.134894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a65aaf-aa0c-4bae-b2b0-6612d2768a4a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:51.135335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a65aaf-aa0c-4bae-b2b0-6612d2768a4a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:51.135437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a65aaf-aa0c-4bae-b2b0-6612d2768a4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:51.154150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f6a65aaf-aa0c-4bae-b2b0-6612d2768a4a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:51.155326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f6a65aaf-aa0c-4bae-b2b0-6612d2768a4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:51.227963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 536/1079] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:51.495846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c3cc87-9c49-480b-b1ca-e207253613f7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:51.497778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607f7a3b-2beb-4c66-85c2-d6f54c1aa70b None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:51.498063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607f7a3b-2beb-4c66-85c2-d6f54c1aa70b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:51.498364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-607f7a3b-2beb-4c66-85c2-d6f54c1aa70b None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:51.499722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 537/1080] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:00:51.505652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80caabfc-1f34-4da6-b3aa-e91db15a911f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:51.505928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80caabfc-1f34-4da6-b3aa-e91db15a911f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:51.506238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80caabfc-1f34-4da6-b3aa-e91db15a911f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:51.526935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-80caabfc-1f34-4da6-b3aa-e91db15a911f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:51.528650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-80caabfc-1f34-4da6-b3aa-e91db15a911f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:51.563327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:51.602280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 538/1081] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:51 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:51.611478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c3cc87-9c49-480b-b1ca-e207253613f7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:51.641460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:51.648568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:51.649203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:51.649475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:51.649743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:51.662212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:51.666930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:51.667972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:51.668817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 544/1082] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:51 2022] GET /identity/v3/roles/admin => generated 83 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:51.674210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80caabfc-1f34-4da6-b3aa-e91db15a911f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:51.776352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80caabfc-1f34-4da6-b3aa-e91db15a911f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:51.810665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:51.811268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:51.811539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:51.811807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:51.812463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:51.812761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:51.814175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:51.822516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 539/1083] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:51 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:51.828530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:51.866932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:51.896083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:51.935508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c394ff-2876-4559-8988-c94b515b67c3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:51.969697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:51.969935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd53e5f8-7540-420e-9b15-db7c4fee7a69 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:51.970349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:51.970622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:51.970892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] PATH_INFO: `/v3/users/c8c8d217f6cc4475997ba7a82434df14` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:51.972043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] RBAC: Authorizing `identity:get_user(user_id=c8c8d217f6cc4475997ba7a82434df14)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:51.974183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:51.975792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 545/1084] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:51 2022] GET /identity/v3/users/c8c8d217f6cc4475997ba7a82434df14 => generated 285 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:51.981675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.006197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.006821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.007120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.007405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.018176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:52.023184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:52.024307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:52.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 540/1085] 10.222.0.61 () {56 vars in 1120 bytes} [Thu Sep 29 02:00:51 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:00:52.030301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.051765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:52.094725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b30287-bc4b-49f7-9dee-b2a4dd2c34c2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:52.116634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:52.129299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c3dcaf-66ff-4db6-8f5e-fbe36a18a2d2 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:52.131213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.131869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.132181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.132456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.133593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] RBAC: Authorizing `identity:get_project(project_id=2a1bc981b8d445fe91329ba5f3e99775)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:52.135052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:52.136898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 546/1086] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:51 2022] GET /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775 => generated 311 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:52.142819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.164090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.164761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.165102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.165401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.166238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:52.166601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:52.167967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:52.176097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 541/1087] 10.222.0.61 () {56 vars in 1115 bytes} [Thu Sep 29 02:00:52 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:52.181994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.243577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3643ebd7-c6ed-424f-ac2b-36097f7778d2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:52.273466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:52.280154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f191981-82e7-4af0-81ef-ea32b1611fde None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:52.283774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.284382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.284743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.284990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775/users/c8c8d217f6cc4475997ba7a82434df14/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.287070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] RBAC: Authorizing `identity:create_grant(project_id=2a1bc981b8d445fe91329ba5f3e99775, user_id=c8c8d217f6cc4475997ba7a82434df14, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:52.288549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:52.302936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 547/1088] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:00:52 2022] PUT /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775/users/c8c8d217f6cc4475997ba7a82434df14/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:00:52.318982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.319653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.319929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.320200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.332474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:52.333973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:52.349803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:52.350916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:52.351812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:52.351812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:52.351812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:52.351812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 542/1089] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:52.357068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.468429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32316843-bcf2-4a74-9eaa-61f43b50b5e5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:52.503056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.504015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.504015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.504207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.504855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:52.505150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:52.506535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:52.518120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 548/1090] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:52.524596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.527724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:52.632522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61d5c3e-9dad-40e5-8be7-5f45c2a07c68 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:52.666664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.667270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.667539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.667803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.675864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:52.677503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:00:52.688917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:52.689973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:52.691071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 543/1091] 10.222.0.61 () {56 vars in 1093 bytes} [Thu Sep 29 02:00:52 2022] GET /identity/v3/users/nova => generated 82 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:52.696188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.746075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:52.803518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c64048d0-2701-4a2c-bc5d-4fddfa810ff9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:52.837938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:52.838558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:52.838825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:52.839089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:52.839801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:52.840099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:52.841512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:52.858660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 549/1092] 10.222.0.61 () {56 vars in 1133 bytes} [Thu Sep 29 02:00:52 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:52.864720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:52.932184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:52.972465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de983b0c-b571-470d-ad8d-435ff4815db9 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:53.007321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:53.007928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:53.008198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:53.008458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:53.076664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:53.078150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:53.137922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:53.182297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 544/1093] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:53.188277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:53.289834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82a99fe-16b3-4895-8c4a-6fc00a6d33ce None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:53.292201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:53.325120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:53.325730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:53.325995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:53.326288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:53.336503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:53.337877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:00:53.353157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:53.354372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:00:53.355567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:00:53.355567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:00:53.355567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:53.358188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 550/1094] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:53.359984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:53.463796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c5bc4dd-e9e6-4a4b-8e04-930c392ad928 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:53.498952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:53.499558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:53.499831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:53.500094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:53.500730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:53.501030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:53.502443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:53.513829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 545/1095] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:00:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:53.519867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:53.625845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a799b29-ea63-4a17-afc6-a6eb77128bf4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:53.660725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:53.661342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:53.661616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:53.661881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:53.668575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:53.670091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:00:53.681965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:53.682955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:00:53.683858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 551/1096] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:00:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:53.689087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:53.715129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:53.797240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bb91ba-c05f-4980-82cf-734568f25d77 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:53.839080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70de620f-5f0e-4a63-85e3-d0fa55d7b0bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:53.839711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de620f-5f0e-4a63-85e3-d0fa55d7b0bd None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:53.839980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de620f-5f0e-4a63-85e3-d0fa55d7b0bd None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:53.840246 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70de620f-5f0e-4a63-85e3-d0fa55d7b0bd None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:53.840930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70de620f-5f0e-4a63-85e3-d0fa55d7b0bd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:53.841226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70de620f-5f0e-4a63-85e3-d0fa55d7b0bd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:53.842702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70de620f-5f0e-4a63-85e3-d0fa55d7b0bd None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:53.854993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 546/1097] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:00:53 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:53.861198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:53.880880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:53.964205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d64af67-4d83-47bd-a536-acc95780531d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:53.998728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be2ac7df-f4a6-408d-8759-caf8aacb32fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:53.999323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ac7df-f4a6-408d-8759-caf8aacb32fb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:53.999592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ac7df-f4a6-408d-8759-caf8aacb32fb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:53.999855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be2ac7df-f4a6-408d-8759-caf8aacb32fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:54.000568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2ac7df-f4a6-408d-8759-caf8aacb32fb None admin] RBAC: Adding query filter params (role.id=3495b485e650453790bdb49f328d4305, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=bc807d5f532b4e4c9953ba5e7eb878bb) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:54.000864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2ac7df-f4a6-408d-8759-caf8aacb32fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:54.002319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be2ac7df-f4a6-408d-8759-caf8aacb32fb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:54.008359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 552/1098] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:53 2022] GET /identity/v3/role_assignments?user.id=bc807d5f532b4e4c9953ba5e7eb878bb&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=3495b485e650453790bdb49f328d4305 => generated 608 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:54.462444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae407307-fe70-43b8-a891-396d6c67b822 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:54.462984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae407307-fe70-43b8-a891-396d6c67b822 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:54.462984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae407307-fe70-43b8-a891-396d6c67b822 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:54.464333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 547/1099] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:54.470140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2dea9a-f780-4163-b274-ced5c3e664ff None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:54.470574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2dea9a-f780-4163-b274-ced5c3e664ff None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:54.470668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2dea9a-f780-4163-b274-ced5c3e664ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:54.490616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6c2dea9a-f780-4163-b274-ced5c3e664ff None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:54.491819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6c2dea9a-f780-4163-b274-ced5c3e664ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:54.571376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 553/1100] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:54 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:54.847072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae407307-fe70-43b8-a891-396d6c67b822 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:54.931545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:54.952635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae407307-fe70-43b8-a891-396d6c67b822 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:54.985497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:54.986135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:54.986405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:54.986667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:54.997285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:00:55.001948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:00:55.003044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:00:55.004390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 548/1101] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:00:54 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:00:55.011950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2dea9a-f780-4163-b274-ced5c3e664ff None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:55.026875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:55.122374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2dea9a-f780-4163-b274-ced5c3e664ff None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:55.158323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:55.158922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:55.159167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:55.159451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:55.160612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:55.160901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:55.162314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:55.170434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 554/1102] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:00:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:55.180622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:55.308647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-539cabd6-d4c2-46c0-a6cf-5976c96f34a3 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:55.345305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:55.345919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:55.346217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:55.346482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] PATH_INFO: `/v3/users/c8c8d217f6cc4475997ba7a82434df14` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:55.347682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] RBAC: Authorizing `identity:get_user(user_id=c8c8d217f6cc4475997ba7a82434df14)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:55.349118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:55.350728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 549/1103] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:00:55 2022] GET /identity/v3/users/c8c8d217f6cc4475997ba7a82434df14 => generated 285 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:55.356856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:55.428556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:55.460737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a01ab2-8fc8-4b1f-9650-c4985b038a39 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:55.495724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:55.496347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:55.496576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:55.496834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:55.566031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:55.567486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:55.635649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:55.666265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 555/1104] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:00:55 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:00:55.673054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:55.686876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:55.778130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7eec6bc-0b8b-4c01-951b-d25aa8cedb48 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:55.814373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24623bd5-fe5e-435a-9131-b9cf93a17906 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:55.814981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24623bd5-fe5e-435a-9131-b9cf93a17906 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:55.815251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24623bd5-fe5e-435a-9131-b9cf93a17906 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:55.815517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24623bd5-fe5e-435a-9131-b9cf93a17906 None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:55.817213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24623bd5-fe5e-435a-9131-b9cf93a17906 None admin] RBAC: Authorizing `identity:get_project(project_id=2a1bc981b8d445fe91329ba5f3e99775)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:55.818683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24623bd5-fe5e-435a-9131-b9cf93a17906 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:55.820275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 550/1105] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:00:55 2022] GET /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775 => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:55.826711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:55.931905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2137ab5-95a8-4841-9372-564daf04d56b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:55.966908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f604a4f-95c8-46c0-bc2c-3047b2fe258c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:55.967427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f604a4f-95c8-46c0-bc2c-3047b2fe258c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:55.967698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f604a4f-95c8-46c0-bc2c-3047b2fe258c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:55.967962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f604a4f-95c8-46c0-bc2c-3047b2fe258c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:55.968671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f604a4f-95c8-46c0-bc2c-3047b2fe258c None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=2a1bc981b8d445fe91329ba5f3e99775, user.id=c8c8d217f6cc4475997ba7a82434df14) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:00:55.968971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f604a4f-95c8-46c0-bc2c-3047b2fe258c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:55.970414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f604a4f-95c8-46c0-bc2c-3047b2fe258c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:55.977453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 556/1106] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:00:55 2022] GET /identity/v3/role_assignments?user.id=c8c8d217f6cc4475997ba7a82434df14&scope.project.id=2a1bc981b8d445fe91329ba5f3e99775&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:58.119472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5f775-9d6e-4093-b682-883788379d57 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:58.120084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5f775-9d6e-4093-b682-883788379d57 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:58.120084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b5f775-9d6e-4093-b682-883788379d57 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:58.122031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 551/1107] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:00:58 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:00:58.130341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd44ede-ef21-41ae-b63b-5c2eeca60cec None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:58.130743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd44ede-ef21-41ae-b63b-5c2eeca60cec None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:58.130882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fd44ede-ef21-41ae-b63b-5c2eeca60cec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:58.149754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9fd44ede-ef21-41ae-b63b-5c2eeca60cec None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:00:58.150992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9fd44ede-ef21-41ae-b63b-5c2eeca60cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:00:58.184318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:58.224160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 557/1108] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:00:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:00:58.491049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b5f775-9d6e-4093-b682-883788379d57 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:58.584528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:00:58.608710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b5f775-9d6e-4093-b682-883788379d57 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:58.644531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3a7907-082f-4b8d-8be8-010b1a8f825b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:58.645156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3a7907-082f-4b8d-8be8-010b1a8f825b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:58.645430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3a7907-082f-4b8d-8be8-010b1a8f825b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:58.645691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3a7907-082f-4b8d-8be8-010b1a8f825b None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:58.646882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3a7907-082f-4b8d-8be8-010b1a8f825b None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:58.648329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3a7907-082f-4b8d-8be8-010b1a8f825b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:58.649895 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 552/1109] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:00:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:00:58.655907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd44ede-ef21-41ae-b63b-5c2eeca60cec None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:00:58.769405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fd44ede-ef21-41ae-b63b-5c2eeca60cec None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:00:58.807253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70fea5c-a352-4662-96e4-13978335f90d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:00:58.807869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70fea5c-a352-4662-96e4-13978335f90d None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:00:58.808144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70fea5c-a352-4662-96e4-13978335f90d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:00:58.808412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70fea5c-a352-4662-96e4-13978335f90d None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:00:58.809073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70fea5c-a352-4662-96e4-13978335f90d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:00:58.810554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70fea5c-a352-4662-96e4-13978335f90d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:00:58.835492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 558/1110] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 02:00:58 2022] POST /identity/v3/users => generated 286 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:01:00.997431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96833432-d7cb-45e1-8177-8d6eb1948b64 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:00.998405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96833432-d7cb-45e1-8177-8d6eb1948b64 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:00.998405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96833432-d7cb-45e1-8177-8d6eb1948b64 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:01.000470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 553/1111] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:00 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:01.006894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d63ddf8-8524-4cf7-b6df-f2f0859dbe70 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:01.007604 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d63ddf8-8524-4cf7-b6df-f2f0859dbe70 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:01.007783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d63ddf8-8524-4cf7-b6df-f2f0859dbe70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:01.031356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9d63ddf8-8524-4cf7-b6df-f2f0859dbe70 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:01.032552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9d63ddf8-8524-4cf7-b6df-f2f0859dbe70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:01.049786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:01.104174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 559/1112] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:01.368922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96833432-d7cb-45e1-8177-8d6eb1948b64 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:01.482966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96833432-d7cb-45e1-8177-8d6eb1948b64 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:01.529054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:01.529689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:01.529961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:01.530423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] PATH_INFO: `/v3/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:01.531930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] RBAC: Authorizing `identity:get_role(role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:01.533321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:01.534756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 554/1113] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:01 2022] GET /identity/v3/roles/1b55a134a7b24115b2bb8691600d2619 => generated 223 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:01.544224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d63ddf8-8524-4cf7-b6df-f2f0859dbe70 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:01.655254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d63ddf8-8524-4cf7-b6df-f2f0859dbe70 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:01.690819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:01.691440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:01.691712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:01.691985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] PATH_INFO: `/v3/users/d2fdcc4923824425a3ddbc09ef9f0345` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:01.705533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] RBAC: Authorizing `identity:get_user(user_id=d2fdcc4923824425a3ddbc09ef9f0345)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:01.707225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:01.708854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 560/1114] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:01 2022] GET /identity/v3/users/d2fdcc4923824425a3ddbc09ef9f0345 => generated 286 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:01.715946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:01.743635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:01.845545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86c1becd-527c-44b1-86d3-33d520a1ab6f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:01.883223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:01.883837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:01.884107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:01.884368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:01.960877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:01.962271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:01.979578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:02.079366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 555/1115] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:01 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:02.085878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:02.102595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:02.217326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a05429a9-1655-497d-80c9-9481f8e59079 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:02.255620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7fa3b17-5dd5-4fcf-9ce7-687aecf11b6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:02.256241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fa3b17-5dd5-4fcf-9ce7-687aecf11b6e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:02.256505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fa3b17-5dd5-4fcf-9ce7-687aecf11b6e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:02.256769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7fa3b17-5dd5-4fcf-9ce7-687aecf11b6e None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:02.258250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fa3b17-5dd5-4fcf-9ce7-687aecf11b6e None admin] RBAC: Authorizing `identity:get_project(project_id=2a1bc981b8d445fe91329ba5f3e99775)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:02.260137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7fa3b17-5dd5-4fcf-9ce7-687aecf11b6e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:02.262487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 561/1116] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:02 2022] GET /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775 => generated 311 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:02.269463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:02.417686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8c523d-e2cc-4375-82c2-c46efbb4847f None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:02.435334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:02.464552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a0a27e-5c11-4b4b-a742-8056e26f06fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:02.465173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a0a27e-5c11-4b4b-a742-8056e26f06fc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:02.465449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a0a27e-5c11-4b4b-a742-8056e26f06fc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:02.465717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a0a27e-5c11-4b4b-a742-8056e26f06fc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:02.466490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a0a27e-5c11-4b4b-a742-8056e26f06fc None admin] RBAC: Adding query filter params (role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=2a1bc981b8d445fe91329ba5f3e99775, user.id=d2fdcc4923824425a3ddbc09ef9f0345) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:02.466799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a0a27e-5c11-4b4b-a742-8056e26f06fc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:02.468334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a0a27e-5c11-4b4b-a742-8056e26f06fc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:02.474368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 556/1117] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:02 2022] GET /identity/v3/role_assignments?user.id=d2fdcc4923824425a3ddbc09ef9f0345&scope.project.id=2a1bc981b8d445fe91329ba5f3e99775&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:04.610676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e539804f-a83a-4f74-b885-b2ada6484365 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:04.611231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e539804f-a83a-4f74-b885-b2ada6484365 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:04.611231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e539804f-a83a-4f74-b885-b2ada6484365 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:04.612688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 562/1118] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:01:04.619217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d48e1a2-dae7-44ec-a4df-598577cb0968 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:04.619942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d48e1a2-dae7-44ec-a4df-598577cb0968 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:04.620127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d48e1a2-dae7-44ec-a4df-598577cb0968 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:04.648983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5d48e1a2-dae7-44ec-a4df-598577cb0968 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:04.650193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5d48e1a2-dae7-44ec-a4df-598577cb0968 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:04.720241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 557/1119] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:04.997804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e539804f-a83a-4f74-b885-b2ada6484365 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:05.088353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:05.116888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e539804f-a83a-4f74-b885-b2ada6484365 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:05.157619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:05.158371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:05.158644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:05.158915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] PATH_INFO: `/v3/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:05.160563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] RBAC: Authorizing `identity:get_role(role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:05.162133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:05.164527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 563/1120] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:04 2022] GET /identity/v3/roles/1b55a134a7b24115b2bb8691600d2619 => generated 223 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:05.172805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d48e1a2-dae7-44ec-a4df-598577cb0968 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:05.275824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d48e1a2-dae7-44ec-a4df-598577cb0968 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:05.285798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:05.322928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:05.323808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:05.324088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:05.324351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] PATH_INFO: `/v3/users/d2fdcc4923824425a3ddbc09ef9f0345` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:05.326252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] RBAC: Authorizing `identity:get_user(user_id=d2fdcc4923824425a3ddbc09ef9f0345)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:05.328460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:05.330903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 558/1121] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:05 2022] GET /identity/v3/users/d2fdcc4923824425a3ddbc09ef9f0345 => generated 286 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:05.340188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:05.449623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010e8f9a-7ea3-42b0-8eaf-b69a7c7a4fd5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:05.495211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af310d0f-fc50-438b-84a4-f299d81cddaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:05.495837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af310d0f-fc50-438b-84a4-f299d81cddaf None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:05.496112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af310d0f-fc50-438b-84a4-f299d81cddaf None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:05.496380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af310d0f-fc50-438b-84a4-f299d81cddaf None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:05.497633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af310d0f-fc50-438b-84a4-f299d81cddaf None admin] RBAC: Authorizing `identity:get_project(project_id=2a1bc981b8d445fe91329ba5f3e99775)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:05.499119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af310d0f-fc50-438b-84a4-f299d81cddaf None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:05.500722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 564/1122] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:05 2022] GET /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775 => generated 311 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:05.506887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:05.621831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a66fd5-f6ca-40f5-9131-54964cfce254 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:05.656684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233bc4de-32a3-4658-9137-3e2f1b75fbbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:05.657330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233bc4de-32a3-4658-9137-3e2f1b75fbbf None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:05.657604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233bc4de-32a3-4658-9137-3e2f1b75fbbf None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:05.657870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233bc4de-32a3-4658-9137-3e2f1b75fbbf None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775/users/d2fdcc4923824425a3ddbc09ef9f0345/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:05.660219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233bc4de-32a3-4658-9137-3e2f1b75fbbf None admin] RBAC: Authorizing `identity:create_grant(project_id=2a1bc981b8d445fe91329ba5f3e99775, user_id=d2fdcc4923824425a3ddbc09ef9f0345, role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:05.661659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233bc4de-32a3-4658-9137-3e2f1b75fbbf None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:05.681898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 559/1123] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:01:05 2022] PUT /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775/users/d2fdcc4923824425a3ddbc09ef9f0345/roles/1b55a134a7b24115b2bb8691600d2619 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:01:07.794718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a3c18a-440e-47ae-8bf4-d60d0e50390a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:07.795273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a3c18a-440e-47ae-8bf4-d60d0e50390a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:07.795273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a3c18a-440e-47ae-8bf4-d60d0e50390a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:07.796593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 565/1124] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:07.803599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e1b63-8962-4961-bbce-35df5c9c81e2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:07.804292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e1b63-8962-4961-bbce-35df5c9c81e2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:07.804445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8e1b63-8962-4961-bbce-35df5c9c81e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:07.833378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1c8e1b63-8962-4961-bbce-35df5c9c81e2 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:07.834597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1c8e1b63-8962-4961-bbce-35df5c9c81e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:07.870486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:07.917520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 560/1125] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:08.193015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a3c18a-440e-47ae-8bf4-d60d0e50390a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:08.211297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:08.304729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a3c18a-440e-47ae-8bf4-d60d0e50390a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:08.339823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:08.340443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:08.340719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:08.340986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] PATH_INFO: `/v3/roles/1b55a134a7b24115b2bb8691600d2619` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:08.342538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] RBAC: Authorizing `identity:get_role(role_id=1b55a134a7b24115b2bb8691600d2619)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:08.343910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:08.345230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 566/1126] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:08 2022] GET /identity/v3/roles/1b55a134a7b24115b2bb8691600d2619 => generated 223 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:08.351604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8e1b63-8962-4961-bbce-35df5c9c81e2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:08.463689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c8e1b63-8962-4961-bbce-35df5c9c81e2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:08.499900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:08.500518 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:08.500794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:08.501066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] PATH_INFO: `/v3/users/d2fdcc4923824425a3ddbc09ef9f0345` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:08.502252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] RBAC: Authorizing `identity:get_user(user_id=d2fdcc4923824425a3ddbc09ef9f0345)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:08.503730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:08.505330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 561/1127] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:08 2022] GET /identity/v3/users/d2fdcc4923824425a3ddbc09ef9f0345 => generated 286 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:08.511520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:08.594816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:08.615681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c71f25d-70e9-4e49-8962-36494692c550 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:08.652985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:08.653575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:08.653852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:08.654151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:08.726584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:08.727972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:08.811885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:08.835650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 567/1128] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:08 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:08.842763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:08.880614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:08.951425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a517980-d04f-4480-8a3e-84a3b3ccd891 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:08.985744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1994d029-1fe4-4f86-b89a-51f50fbfc21c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:08.986342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1994d029-1fe4-4f86-b89a-51f50fbfc21c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:08.986614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1994d029-1fe4-4f86-b89a-51f50fbfc21c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:08.986879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1994d029-1fe4-4f86-b89a-51f50fbfc21c None admin] PATH_INFO: `/v3/projects/2a1bc981b8d445fe91329ba5f3e99775` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:08.988005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1994d029-1fe4-4f86-b89a-51f50fbfc21c None admin] RBAC: Authorizing `identity:get_project(project_id=2a1bc981b8d445fe91329ba5f3e99775)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:08.989656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1994d029-1fe4-4f86-b89a-51f50fbfc21c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:08.991311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 562/1129] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:08 2022] GET /identity/v3/projects/2a1bc981b8d445fe91329ba5f3e99775 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:08.998294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:09.104902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde1bf6a-8691-4d55-ae87-5865667ddc0b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:09.145572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ffda43-0f6d-4458-b348-e791aeb8321e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:09.146204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ffda43-0f6d-4458-b348-e791aeb8321e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:09.146483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ffda43-0f6d-4458-b348-e791aeb8321e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:09.146757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ffda43-0f6d-4458-b348-e791aeb8321e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:09.147510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ffda43-0f6d-4458-b348-e791aeb8321e None admin] RBAC: Adding query filter params (role.id=1b55a134a7b24115b2bb8691600d2619, scope.project.id=2a1bc981b8d445fe91329ba5f3e99775, user.id=d2fdcc4923824425a3ddbc09ef9f0345) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:09.147818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ffda43-0f6d-4458-b348-e791aeb8321e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:09.149253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ffda43-0f6d-4458-b348-e791aeb8321e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:09.155265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 568/1130] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:08 2022] GET /identity/v3/role_assignments?user.id=d2fdcc4923824425a3ddbc09ef9f0345&scope.project.id=2a1bc981b8d445fe91329ba5f3e99775&role.id=1b55a134a7b24115b2bb8691600d2619 => generated 608 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:11.339813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efe5e9-5dab-4c0e-9541-6fb596558526 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:11.340369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efe5e9-5dab-4c0e-9541-6fb596558526 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:11.340369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64efe5e9-5dab-4c0e-9541-6fb596558526 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:11.342497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 563/1131] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:11 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:11.351506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bd362-3149-4d46-bd3b-83d7e97f1ebd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:11.352257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bd362-3149-4d46-bd3b-83d7e97f1ebd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:11.352426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7bd362-3149-4d46-bd3b-83d7e97f1ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:11.375892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cc7bd362-3149-4d46-bd3b-83d7e97f1ebd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:11.377072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cc7bd362-3149-4d46-bd3b-83d7e97f1ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:11.421645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:11.452162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 569/1132] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:11.721218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efe5e9-5dab-4c0e-9541-6fb596558526 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:11.837763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64efe5e9-5dab-4c0e-9541-6fb596558526 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:11.839397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:11.873966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdfcf29-5aff-4804-8d68-644a62e9d572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:11.874600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfcf29-5aff-4804-8d68-644a62e9d572 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:11.874869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfcf29-5aff-4804-8d68-644a62e9d572 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:11.875135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdfcf29-5aff-4804-8d68-644a62e9d572 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:11.876231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdfcf29-5aff-4804-8d68-644a62e9d572 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:11.877664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdfcf29-5aff-4804-8d68-644a62e9d572 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:11.879330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 564/1133] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:11.886382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bd362-3149-4d46-bd3b-83d7e97f1ebd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:11.997770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7bd362-3149-4d46-bd3b-83d7e97f1ebd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:12.033392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40bf0189-9e73-494c-8b50-0e60931fc789 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:12.033987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bf0189-9e73-494c-8b50-0e60931fc789 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:12.034287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bf0189-9e73-494c-8b50-0e60931fc789 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:12.034556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40bf0189-9e73-494c-8b50-0e60931fc789 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:12.035223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bf0189-9e73-494c-8b50-0e60931fc789 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:12.036738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40bf0189-9e73-494c-8b50-0e60931fc789 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:12.067210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 570/1134] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:01:11 2022] POST /identity/v3/projects => generated 311 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:01:14.244299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354331d4-032d-472b-8fb2-ccc1aeafd88d None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:14.245212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354331d4-032d-472b-8fb2-ccc1aeafd88d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:14.245212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354331d4-032d-472b-8fb2-ccc1aeafd88d None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:14.247338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 565/1135] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:14 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:14.253957 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b120ddde-2762-4840-a5f5-b30601d4bf12 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:14.255131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b120ddde-2762-4840-a5f5-b30601d4bf12 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:14.255599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b120ddde-2762-4840-a5f5-b30601d4bf12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:14.280597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b120ddde-2762-4840-a5f5-b30601d4bf12 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:14.281774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b120ddde-2762-4840-a5f5-b30601d4bf12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:14.306978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:14.361504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 571/1136] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:14.630404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354331d4-032d-472b-8fb2-ccc1aeafd88d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:14.761100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354331d4-032d-472b-8fb2-ccc1aeafd88d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:14.799798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efaa3aab-1d80-498a-9ba7-3d169955077d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:14.800355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaa3aab-1d80-498a-9ba7-3d169955077d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:14.800633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaa3aab-1d80-498a-9ba7-3d169955077d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:14.800868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efaa3aab-1d80-498a-9ba7-3d169955077d None admin] PATH_INFO: `/v3/domains/default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:14.802585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaa3aab-1d80-498a-9ba7-3d169955077d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:14.804039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efaa3aab-1d80-498a-9ba7-3d169955077d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:14.805614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 566/1137] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:01:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:14.814406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b120ddde-2762-4840-a5f5-b30601d4bf12 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:14.927805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b120ddde-2762-4840-a5f5-b30601d4bf12 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:14.965577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5110963-d044-4c0f-abb5-20b0ab9afaeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:14.966198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5110963-d044-4c0f-abb5-20b0ab9afaeb None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:14.966467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5110963-d044-4c0f-abb5-20b0ab9afaeb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:14.966735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5110963-d044-4c0f-abb5-20b0ab9afaeb None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:14.967388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5110963-d044-4c0f-abb5-20b0ab9afaeb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:14.968821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5110963-d044-4c0f-abb5-20b0ab9afaeb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:14.995053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 572/1138] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 02:01:14 2022] POST /identity/v3/users => generated 286 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:01:17.182849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad978b6-f967-471e-9bc1-638d535bf3b4 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:17.183722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad978b6-f967-471e-9bc1-638d535bf3b4 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:17.183722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aad978b6-f967-471e-9bc1-638d535bf3b4 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:17.185890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 567/1139] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:17.193873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6b550-b871-4bd4-9249-8c909b8fab5d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:17.194360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6b550-b871-4bd4-9249-8c909b8fab5d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:17.194459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb6b550-b871-4bd4-9249-8c909b8fab5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:17.218725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fcb6b550-b871-4bd4-9249-8c909b8fab5d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:17.219950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fcb6b550-b871-4bd4-9249-8c909b8fab5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:17.231019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:17.295910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 573/1140] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:17.560339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad978b6-f967-471e-9bc1-638d535bf3b4 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:17.606771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:17.673712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aad978b6-f967-471e-9bc1-638d535bf3b4 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:17.710332 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:17.710955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:17.711230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:17.711499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:17.722234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:17.727033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:17.728163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:01:17.729207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 568/1141] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:17.737148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6b550-b871-4bd4-9249-8c909b8fab5d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:17.843055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb6b550-b871-4bd4-9249-8c909b8fab5d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:17.877494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:17.878106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:17.878375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:17.878643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:17.879288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:17.879627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:17.881025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:17.888856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 574/1142] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:17.895610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:18.001366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:18.005214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-868aa221-3dcd-4b79-9b6b-a9f37c2dbe38 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:18.044395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:18.045014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:18.045289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:18.045557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] PATH_INFO: `/v3/users/8c3e7aa73e78406ebf95233d0082ed01` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:18.060555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] RBAC: Authorizing `identity:get_user(user_id=8c3e7aa73e78406ebf95233d0082ed01)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:18.062030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:18.063612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 569/1143] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:17 2022] GET /identity/v3/users/8c3e7aa73e78406ebf95233d0082ed01 => generated 286 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:18.073236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:18.086232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:18.182255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c574b76-8e16-4a2c-80ca-89fa93fda89f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:18.216637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:18.217268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:18.217471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:18.217738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:18.303358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:18.304800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:18.312315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:18.406374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 575/1144] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:18 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:18.412767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:18.515443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad29ae6a-3cfa-4bd6-8871-f2ff8c7fd5a7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:18.555434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e5b1e2-7ad6-4c22-b82c-6cf642d0cdef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:18.556064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e5b1e2-7ad6-4c22-b82c-6cf642d0cdef None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:18.556315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e5b1e2-7ad6-4c22-b82c-6cf642d0cdef None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:18.556582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e5b1e2-7ad6-4c22-b82c-6cf642d0cdef None admin] PATH_INFO: `/v3/projects/ddea876dda404499838210cec6c42df0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:18.557771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e5b1e2-7ad6-4c22-b82c-6cf642d0cdef None admin] RBAC: Authorizing `identity:get_project(project_id=ddea876dda404499838210cec6c42df0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:18.559239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e5b1e2-7ad6-4c22-b82c-6cf642d0cdef None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:18.560857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 570/1145] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:18 2022] GET /identity/v3/projects/ddea876dda404499838210cec6c42df0 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:18.567497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:18.672890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14dfb1ed-cf99-4157-93a6-90eb8b623b73 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:18.680496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:18.712090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ceabeeb-f710-4682-82a8-c09be5037a7e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:18.712687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceabeeb-f710-4682-82a8-c09be5037a7e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:18.712960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceabeeb-f710-4682-82a8-c09be5037a7e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:18.713229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ceabeeb-f710-4682-82a8-c09be5037a7e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:18.713950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceabeeb-f710-4682-82a8-c09be5037a7e None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ddea876dda404499838210cec6c42df0, user.id=8c3e7aa73e78406ebf95233d0082ed01) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:18.714304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceabeeb-f710-4682-82a8-c09be5037a7e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:18.715740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ceabeeb-f710-4682-82a8-c09be5037a7e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:18.721555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 576/1146] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:18 2022] GET /identity/v3/role_assignments?user.id=8c3e7aa73e78406ebf95233d0082ed01&scope.project.id=ddea876dda404499838210cec6c42df0&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:20.840273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4c4aa6-ba63-4518-8310-ab62c175411d None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:20.840842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4c4aa6-ba63-4518-8310-ab62c175411d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:20.840842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4c4aa6-ba63-4518-8310-ab62c175411d None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:20.842942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 571/1147] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:20.852066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9112d64d-0954-4dee-88d4-15dc16b9b8bf None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:20.852525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9112d64d-0954-4dee-88d4-15dc16b9b8bf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:20.852642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9112d64d-0954-4dee-88d4-15dc16b9b8bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:20.882727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9112d64d-0954-4dee-88d4-15dc16b9b8bf None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:20.883922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9112d64d-0954-4dee-88d4-15dc16b9b8bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:20.957606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 577/1148] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:21.241606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4c4aa6-ba63-4518-8310-ab62c175411d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:21.280482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:21.368216 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4c4aa6-ba63-4518-8310-ab62c175411d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:21.407661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:21.408280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:21.408551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:21.408839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:21.419809 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:21.424574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:21.425681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:01:21.427454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 572/1149] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:21 2022] GET /identity/v3/roles/admin => generated 83 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:21.434500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9112d64d-0954-4dee-88d4-15dc16b9b8bf None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:21.540370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:21.543632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9112d64d-0954-4dee-88d4-15dc16b9b8bf None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:21.581748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:21.582386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:21.582679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:21.583386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:21.584085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:21.584385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:21.585753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:21.594684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 578/1150] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:21 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:21.601543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:21.701337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:21.719577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03dbd429-4a01-4402-8590-cee1f5466db9 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:21.756243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:21.756863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:21.757137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:21.757405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] PATH_INFO: `/v3/users/8c3e7aa73e78406ebf95233d0082ed01` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:21.758663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] RBAC: Authorizing `identity:get_user(user_id=8c3e7aa73e78406ebf95233d0082ed01)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:21.760110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:21.761684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 573/1151] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:21 2022] GET /identity/v3/users/8c3e7aa73e78406ebf95233d0082ed01 => generated 286 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:21.771116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:21.874677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7ce39f-a62f-48af-a9b2-d2cff1be8a52 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:21.911519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dd25391-2360-447d-b892-960d469c730b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:21.912158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd25391-2360-447d-b892-960d469c730b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:21.912430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd25391-2360-447d-b892-960d469c730b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:21.912696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dd25391-2360-447d-b892-960d469c730b None admin] PATH_INFO: `/v3/projects/ddea876dda404499838210cec6c42df0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:21.913855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd25391-2360-447d-b892-960d469c730b None admin] RBAC: Authorizing `identity:get_project(project_id=ddea876dda404499838210cec6c42df0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:21.915308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dd25391-2360-447d-b892-960d469c730b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:21.916920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 579/1152] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:21 2022] GET /identity/v3/projects/ddea876dda404499838210cec6c42df0 => generated 311 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:21.923881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:22.033031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-242cab6b-4a2b-4a8d-bf90-0e682e008899 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:22.077327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef44ad0-422e-4334-a98b-b06bcc607664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:22.077990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef44ad0-422e-4334-a98b-b06bcc607664 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:22.078290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef44ad0-422e-4334-a98b-b06bcc607664 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:22.078560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef44ad0-422e-4334-a98b-b06bcc607664 None admin] PATH_INFO: `/v3/projects/ddea876dda404499838210cec6c42df0/users/8c3e7aa73e78406ebf95233d0082ed01/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:22.080708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef44ad0-422e-4334-a98b-b06bcc607664 None admin] RBAC: Authorizing `identity:create_grant(project_id=ddea876dda404499838210cec6c42df0, user_id=8c3e7aa73e78406ebf95233d0082ed01, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:22.082171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef44ad0-422e-4334-a98b-b06bcc607664 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:22.101921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 574/1153] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:01:21 2022] PUT /identity/v3/projects/ddea876dda404499838210cec6c42df0/users/8c3e7aa73e78406ebf95233d0082ed01/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:01:24.204460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00201f1f-c4a9-4a4e-a52f-4192b0553207 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:24.205127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00201f1f-c4a9-4a4e-a52f-4192b0553207 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:24.205127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00201f1f-c4a9-4a4e-a52f-4192b0553207 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:24.206378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 580/1154] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:24.213027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b989dd1e-b6ec-47e1-af6a-38dfd87582f4 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:24.213794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b989dd1e-b6ec-47e1-af6a-38dfd87582f4 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:24.214167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b989dd1e-b6ec-47e1-af6a-38dfd87582f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:24.242143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b989dd1e-b6ec-47e1-af6a-38dfd87582f4 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:24.243345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b989dd1e-b6ec-47e1-af6a-38dfd87582f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:24.250545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:24.321262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 575/1155] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:24 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:24.586342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00201f1f-c4a9-4a4e-a52f-4192b0553207 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:24.622859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:24.702216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00201f1f-c4a9-4a4e-a52f-4192b0553207 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:24.737590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:24.738227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:24.738499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:24.738767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:24.749953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:24.754715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:24.756099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:01:24.757284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 581/1156] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:24 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:24.762091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b989dd1e-b6ec-47e1-af6a-38dfd87582f4 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:24.862456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b989dd1e-b6ec-47e1-af6a-38dfd87582f4 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:24.897946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:24.898564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:24.898835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:24.899097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:24.899745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:24.900036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:24.901413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:24.909602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 576/1157] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:24.919276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:25.023257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:25.033543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c2f203-0276-4ef4-bb5f-4519306a2022 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:25.069112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:25.069734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:25.070023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:25.070308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] PATH_INFO: `/v3/users/8c3e7aa73e78406ebf95233d0082ed01` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:25.071415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] RBAC: Authorizing `identity:get_user(user_id=8c3e7aa73e78406ebf95233d0082ed01)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:25.072846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:25.074455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 582/1158] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:24 2022] GET /identity/v3/users/8c3e7aa73e78406ebf95233d0082ed01 => generated 286 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:25.080458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:25.084480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:25.186500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad97aeb5-1076-454f-80d8-8cc7e27acb5c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:25.221939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:25.222903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:25.223250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:25.223748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:25.291693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:25.293126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:25.295908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:25.398142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 577/1159] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:25 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:25.404870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:25.513019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11526dff-abe1-4cbe-a96e-d85e6075a34c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:25.548578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed567c52-3500-40f4-b85c-5657a1956ff7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:25.549209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed567c52-3500-40f4-b85c-5657a1956ff7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:25.549485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed567c52-3500-40f4-b85c-5657a1956ff7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:25.549756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed567c52-3500-40f4-b85c-5657a1956ff7 None admin] PATH_INFO: `/v3/projects/ddea876dda404499838210cec6c42df0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:25.550962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed567c52-3500-40f4-b85c-5657a1956ff7 None admin] RBAC: Authorizing `identity:get_project(project_id=ddea876dda404499838210cec6c42df0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:25.552832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed567c52-3500-40f4-b85c-5657a1956ff7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:25.554520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 583/1160] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:25 2022] GET /identity/v3/projects/ddea876dda404499838210cec6c42df0 => generated 311 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:25.561608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:25.682461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af1d88e-60ab-48b8-bd19-ad5314b8fcc3 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:25.684951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:25.720134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae2867db-f399-478a-8038-ccdc151e691c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:25.720727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2867db-f399-478a-8038-ccdc151e691c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:25.720998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2867db-f399-478a-8038-ccdc151e691c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:25.721257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae2867db-f399-478a-8038-ccdc151e691c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:25.721967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2867db-f399-478a-8038-ccdc151e691c None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ddea876dda404499838210cec6c42df0, user.id=8c3e7aa73e78406ebf95233d0082ed01) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:25.722299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2867db-f399-478a-8038-ccdc151e691c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:25.724070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae2867db-f399-478a-8038-ccdc151e691c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:25.729820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 578/1161] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:25 2022] GET /identity/v3/role_assignments?user.id=8c3e7aa73e78406ebf95233d0082ed01&scope.project.id=ddea876dda404499838210cec6c42df0&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:27.864695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c6749c-0140-4e1b-8f0d-c3e6f7b51e3a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:27.865284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c6749c-0140-4e1b-8f0d-c3e6f7b51e3a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:27.865284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c6749c-0140-4e1b-8f0d-c3e6f7b51e3a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:27.866601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 584/1162] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:27.872848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bb0d9d-4cf4-4d29-adef-de7f0924e86a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:27.873579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bb0d9d-4cf4-4d29-adef-de7f0924e86a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:27.873758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bb0d9d-4cf4-4d29-adef-de7f0924e86a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:27.901175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-47bb0d9d-4cf4-4d29-adef-de7f0924e86a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:27.902385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-47bb0d9d-4cf4-4d29-adef-de7f0924e86a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:27.971909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 579/1163] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:28.241514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c6749c-0140-4e1b-8f0d-c3e6f7b51e3a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:28.271916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:28.350523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c6749c-0140-4e1b-8f0d-c3e6f7b51e3a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:28.385382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:28.385977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:28.386284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:28.386555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:28.455084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:28.456531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:28.482869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:28.561544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 585/1164] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:28 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:28.567839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bb0d9d-4cf4-4d29-adef-de7f0924e86a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:28.665951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:28.671355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47bb0d9d-4cf4-4d29-adef-de7f0924e86a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:28.709047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6852e81-ddf5-4e47-b595-0ac2235392ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:28.709665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6852e81-ddf5-4e47-b595-0ac2235392ea None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:28.709936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6852e81-ddf5-4e47-b595-0ac2235392ea None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:28.710226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6852e81-ddf5-4e47-b595-0ac2235392ea None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:28.716818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6852e81-ddf5-4e47-b595-0ac2235392ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:28.718300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6852e81-ddf5-4e47-b595-0ac2235392ea None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-e6852e81-ddf5-4e47-b595-0ac2235392ea None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:28.730039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:28.731203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:01:28.732327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 580/1165] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:01:28 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:28.738880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:28.841918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f834834-a12f-4aa6-a239-4990bf6441dc None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:28.867375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:28.877497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552d771f-4ab9-4938-adb6-2ba29697f6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:28.878113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d771f-4ab9-4938-adb6-2ba29697f6c9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:28.878383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d771f-4ab9-4938-adb6-2ba29697f6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:28.878648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552d771f-4ab9-4938-adb6-2ba29697f6c9 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:28.879279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552d771f-4ab9-4938-adb6-2ba29697f6c9 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:28.879570 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552d771f-4ab9-4938-adb6-2ba29697f6c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:28.881005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552d771f-4ab9-4938-adb6-2ba29697f6c9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:28.888127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 586/1166] 10.222.0.61 () {56 vars in 1115 bytes} [Thu Sep 29 02:01:28 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:30.992055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205c5829-365b-4f58-9d18-204f10627764 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:30.992913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205c5829-365b-4f58-9d18-204f10627764 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:30.992913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-205c5829-365b-4f58-9d18-204f10627764 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:30.995094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 581/1167] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:31.003092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115fb3f7-4fbe-429e-9f04-915f2997db49 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:31.003547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115fb3f7-4fbe-429e-9f04-915f2997db49 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:31.003645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-115fb3f7-4fbe-429e-9f04-915f2997db49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:31.023881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-115fb3f7-4fbe-429e-9f04-915f2997db49 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:31.025043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-115fb3f7-4fbe-429e-9f04-915f2997db49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:31.093339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 587/1168] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:31 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:31.356143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205c5829-365b-4f58-9d18-204f10627764 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:31.468668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-205c5829-365b-4f58-9d18-204f10627764 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:31.512846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8af48de-dfb2-4707-b9c7-c78bd59b2e0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:31.513439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8af48de-dfb2-4707-b9c7-c78bd59b2e0c None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:31.513712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8af48de-dfb2-4707-b9c7-c78bd59b2e0c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:31.513983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8af48de-dfb2-4707-b9c7-c78bd59b2e0c None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:31.514689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8af48de-dfb2-4707-b9c7-c78bd59b2e0c None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:31.516097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8af48de-dfb2-4707-b9c7-c78bd59b2e0c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:31.538163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 582/1169] 10.222.0.61 () {60 vars in 1138 bytes} [Thu Sep 29 02:01:31 2022] POST /identity/v3/domains => generated 267 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:01:33.686871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae189714-f232-43f1-876f-fc9fd11a824a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:33.687756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae189714-f232-43f1-876f-fc9fd11a824a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:33.687756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae189714-f232-43f1-876f-fc9fd11a824a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:33.689826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 588/1170] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:33.699000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837572e8-aba4-4067-8e6e-6e28495933fe None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:33.699437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837572e8-aba4-4067-8e6e-6e28495933fe None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:33.699600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837572e8-aba4-4067-8e6e-6e28495933fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:33.722154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-837572e8-aba4-4067-8e6e-6e28495933fe None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:33.723342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-837572e8-aba4-4067-8e6e-6e28495933fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:33.737269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:33.805716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 583/1171] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:34.071499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae189714-f232-43f1-876f-fc9fd11a824a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:34.177755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae189714-f232-43f1-876f-fc9fd11a824a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:34.198030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:34.215220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2005961-2330-422c-8c6f-cd509d7a922c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:34.215834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2005961-2330-422c-8c6f-cd509d7a922c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:34.216101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2005961-2330-422c-8c6f-cd509d7a922c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:34.216369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2005961-2330-422c-8c6f-cd509d7a922c None admin] PATH_INFO: `/v3/domains/75b4a28ec80a40c59c37898a94585146` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:34.228634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2005961-2330-422c-8c6f-cd509d7a922c None admin] RBAC: Authorizing `identity:get_domain(domain_id=75b4a28ec80a40c59c37898a94585146)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:34.230059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2005961-2330-422c-8c6f-cd509d7a922c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:34.231656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 589/1172] 10.222.0.61 () {56 vars in 1183 bytes} [Thu Sep 29 02:01:34 2022] GET /identity/v3/domains/75b4a28ec80a40c59c37898a94585146 => generated 267 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:34.238311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837572e8-aba4-4067-8e6e-6e28495933fe None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:34.350758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837572e8-aba4-4067-8e6e-6e28495933fe None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:34.385425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e016cfde-be67-45a6-9ffd-5f4a0c79e6dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:34.386045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016cfde-be67-45a6-9ffd-5f4a0c79e6dc None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:34.386312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016cfde-be67-45a6-9ffd-5f4a0c79e6dc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:34.386577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e016cfde-be67-45a6-9ffd-5f4a0c79e6dc None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:34.387213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e016cfde-be67-45a6-9ffd-5f4a0c79e6dc None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:34.388651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e016cfde-be67-45a6-9ffd-5f4a0c79e6dc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:34.412413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:34.418705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 584/1173] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:01:34 2022] POST /identity/v3/projects => generated 361 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:01:36.593327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821fea37-ea10-45bc-85bf-79e101312134 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:36.594170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821fea37-ea10-45bc-85bf-79e101312134 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:36.594170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821fea37-ea10-45bc-85bf-79e101312134 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:36.596340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 590/1174] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:36 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:36.604189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4b4f1-64b7-481f-8284-8cc94ad17319 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:36.604613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4b4f1-64b7-481f-8284-8cc94ad17319 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:36.604752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4b4f1-64b7-481f-8284-8cc94ad17319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:36.633033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-84e4b4f1-64b7-481f-8284-8cc94ad17319 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:36.634256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-84e4b4f1-64b7-481f-8284-8cc94ad17319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:36.705081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 585/1175] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:36 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:36.969446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821fea37-ea10-45bc-85bf-79e101312134 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:37.077106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821fea37-ea10-45bc-85bf-79e101312134 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:37.110254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc4992e-71f4-40da-a421-c243f35e2b92 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:37.110889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc4992e-71f4-40da-a421-c243f35e2b92 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:37.111152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc4992e-71f4-40da-a421-c243f35e2b92 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:37.111419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc4992e-71f4-40da-a421-c243f35e2b92 None admin] PATH_INFO: `/v3/domains/75b4a28ec80a40c59c37898a94585146` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:37.112644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc4992e-71f4-40da-a421-c243f35e2b92 None admin] RBAC: Authorizing `identity:get_domain(domain_id=75b4a28ec80a40c59c37898a94585146)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:37.114075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc4992e-71f4-40da-a421-c243f35e2b92 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:37.115653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 591/1176] 10.222.0.61 () {56 vars in 1183 bytes} [Thu Sep 29 02:01:36 2022] GET /identity/v3/domains/75b4a28ec80a40c59c37898a94585146 => generated 267 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:37.121730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e4b4f1-64b7-481f-8284-8cc94ad17319 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:37.228571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e4b4f1-64b7-481f-8284-8cc94ad17319 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:37.258025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:37.265760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff4cae84-d80d-4359-bb95-55594c13d13b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:37.266399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4cae84-d80d-4359-bb95-55594c13d13b None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:37.266673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4cae84-d80d-4359-bb95-55594c13d13b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:37.266942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff4cae84-d80d-4359-bb95-55594c13d13b None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:37.267585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4cae84-d80d-4359-bb95-55594c13d13b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:37.269230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff4cae84-d80d-4359-bb95-55594c13d13b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:37.293058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 586/1177] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 02:01:37 2022] POST /identity/v3/users => generated 311 bytes in 172 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:01:39.505680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eca272-acdf-4dbf-b8ab-53bf447337bd None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:39.506344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eca272-acdf-4dbf-b8ab-53bf447337bd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:39.506344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24eca272-acdf-4dbf-b8ab-53bf447337bd None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:39.508487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 592/1178] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:39.517333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707251a7-856c-46c4-9ef8-8df8e2fb30c4 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:39.517743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707251a7-856c-46c4-9ef8-8df8e2fb30c4 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:39.517924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707251a7-856c-46c4-9ef8-8df8e2fb30c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:39.545056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-707251a7-856c-46c4-9ef8-8df8e2fb30c4 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:39.546293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-707251a7-856c-46c4-9ef8-8df8e2fb30c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:39.616270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 587/1179] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:39.881691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eca272-acdf-4dbf-b8ab-53bf447337bd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:39.941273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:39.997228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24eca272-acdf-4dbf-b8ab-53bf447337bd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.033164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.033769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.034045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.034331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.045485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:40.050196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:40.051616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:01:40.052608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 593/1180] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:40.059021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707251a7-856c-46c4-9ef8-8df8e2fb30c4 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.158157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707251a7-856c-46c4-9ef8-8df8e2fb30c4 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.181346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.205869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.206505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.206773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.207036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.207684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:40.207979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.209443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.217223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 588/1181] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:40.223571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.333326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d50e9dc-502d-440f-b39d-308866818a0d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.348116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.375531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.376151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.376423 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.376694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] PATH_INFO: `/v3/users/b07e7fa4fbae41bcb4856638c0dbcdce` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.390502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] RBAC: Authorizing `identity:get_user(user_id=b07e7fa4fbae41bcb4856638c0dbcdce)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.391941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.393534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 594/1182] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/users/b07e7fa4fbae41bcb4856638c0dbcdce => generated 311 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:40.399867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.505406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6908f1-aaa8-4808-a84f-7a3c8464face None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.547779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.548765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.549235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.550103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.599867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:40.626103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.627573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.741254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 589/1183] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 342 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:40.750989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.857873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3bba8a0-0b4c-4fbe-b492-7d3b5df3cf13 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:40.897732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7fadc6-1035-47b3-8311-eb13202bae1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:40.898368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7fadc6-1035-47b3-8311-eb13202bae1f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:40.898638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7fadc6-1035-47b3-8311-eb13202bae1f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:40.898899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7fadc6-1035-47b3-8311-eb13202bae1f None admin] PATH_INFO: `/v3/projects/7e92e97b91e04df18fece2fc5ce713e8` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:40.900121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7fadc6-1035-47b3-8311-eb13202bae1f None admin] RBAC: Authorizing `identity:get_project(project_id=7e92e97b91e04df18fece2fc5ce713e8)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:40.901546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7fadc6-1035-47b3-8311-eb13202bae1f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:40.903184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 595/1184] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/projects/7e92e97b91e04df18fece2fc5ce713e8 => generated 361 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:40.910080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:40.997677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:41.019708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56f75a4-4814-4f5a-ac56-8bba4c5a970e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:41.052753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4453791a-b787-42b2-83d6-dfe83b799994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:41.053397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453791a-b787-42b2-83d6-dfe83b799994 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:41.053674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453791a-b787-42b2-83d6-dfe83b799994 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:41.053976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4453791a-b787-42b2-83d6-dfe83b799994 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:41.054743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4453791a-b787-42b2-83d6-dfe83b799994 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=7e92e97b91e04df18fece2fc5ce713e8, user.id=b07e7fa4fbae41bcb4856638c0dbcdce) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:41.055044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4453791a-b787-42b2-83d6-dfe83b799994 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:41.056466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4453791a-b787-42b2-83d6-dfe83b799994 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:41.062994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 590/1185] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:40 2022] GET /identity/v3/role_assignments?user.id=b07e7fa4fbae41bcb4856638c0dbcdce&scope.project.id=7e92e97b91e04df18fece2fc5ce713e8&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:43.164721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53087c9-edcb-458f-a880-1d880cda0261 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.165281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53087c9-edcb-458f-a880-1d880cda0261 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.165281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53087c9-edcb-458f-a880-1d880cda0261 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.167394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 596/1186] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:43 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:43.175437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d480fd4-f626-4ab0-a017-d2961a1a4c5b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.175910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d480fd4-f626-4ab0-a017-d2961a1a4c5b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.175910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d480fd4-f626-4ab0-a017-d2961a1a4c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.201444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6d480fd4-f626-4ab0-a017-d2961a1a4c5b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:43.202658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6d480fd4-f626-4ab0-a017-d2961a1a4c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:43.270371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 591/1187] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:43 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:01:43.540288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53087c9-edcb-458f-a880-1d880cda0261 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:43.582213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:43.668870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53087c9-edcb-458f-a880-1d880cda0261 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:43.705166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:43.705769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.706040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.706330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.717212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:43.721930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:43.723049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:01:43.724053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 597/1188] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:43 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:43.729289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d480fd4-f626-4ab0-a017-d2961a1a4c5b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:43.799707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:43.841451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d480fd4-f626-4ab0-a017-d2961a1a4c5b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:43.877847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:43.878476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:43.878753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:43.879018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:43.879671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:43.879973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:43.881451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:43.889862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 592/1189] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:43 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:43.900153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.007050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:44.014843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dccd786-1c83-4080-b5c4-603a0f9aa2db None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.049850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.050427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.050701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.050971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] PATH_INFO: `/v3/users/b07e7fa4fbae41bcb4856638c0dbcdce` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.052487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] RBAC: Authorizing `identity:get_user(user_id=b07e7fa4fbae41bcb4856638c0dbcdce)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.054568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.056161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 598/1190] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:43 2022] GET /identity/v3/users/b07e7fa4fbae41bcb4856638c0dbcdce => generated 311 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:44.064779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.173913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7e1e4d-1c68-4730-9fee-2b61806e3f8c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.211110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67471acc-fec2-4b59-aaa7-be158dd301ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.211738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67471acc-fec2-4b59-aaa7-be158dd301ca None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.211978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67471acc-fec2-4b59-aaa7-be158dd301ca None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.212243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67471acc-fec2-4b59-aaa7-be158dd301ca None admin] PATH_INFO: `/v3/projects/7e92e97b91e04df18fece2fc5ce713e8` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.213394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67471acc-fec2-4b59-aaa7-be158dd301ca None admin] RBAC: Authorizing `identity:get_project(project_id=7e92e97b91e04df18fece2fc5ce713e8)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.214838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67471acc-fec2-4b59-aaa7-be158dd301ca None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.216467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 593/1191] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:44 2022] GET /identity/v3/projects/7e92e97b91e04df18fece2fc5ce713e8 => generated 361 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:44.226404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:44.338775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7173b85c-6a78-4956-974a-387b8f763c84 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:44.373799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-797a6535-34e6-4dca-bc15-6a097bcc1500 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:44.374465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797a6535-34e6-4dca-bc15-6a097bcc1500 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:44.374730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797a6535-34e6-4dca-bc15-6a097bcc1500 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:44.374991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-797a6535-34e6-4dca-bc15-6a097bcc1500 None admin] PATH_INFO: `/v3/projects/7e92e97b91e04df18fece2fc5ce713e8/users/b07e7fa4fbae41bcb4856638c0dbcdce/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:44.377109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797a6535-34e6-4dca-bc15-6a097bcc1500 None admin] RBAC: Authorizing `identity:create_grant(project_id=7e92e97b91e04df18fece2fc5ce713e8, user_id=b07e7fa4fbae41bcb4856638c0dbcdce, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:44.378567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-797a6535-34e6-4dca-bc15-6a097bcc1500 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:44.395101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 599/1192] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:01:44 2022] PUT /identity/v3/projects/7e92e97b91e04df18fece2fc5ce713e8/users/b07e7fa4fbae41bcb4856638c0dbcdce/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:01:46.551686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e56ca-3624-45f0-8ca0-27f884e5c210 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:46.552125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e56ca-3624-45f0-8ca0-27f884e5c210 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:46.552231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35e56ca-3624-45f0-8ca0-27f884e5c210 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:46.553780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 594/1193] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:01:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:01:46.559320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd3253e-8ab3-4bbd-8869-06660c686abb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:46.559763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd3253e-8ab3-4bbd-8869-06660c686abb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:46.559864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddd3253e-8ab3-4bbd-8869-06660c686abb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:46.579562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ddd3253e-8ab3-4bbd-8869-06660c686abb None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:01:46.580756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ddd3253e-8ab3-4bbd-8869-06660c686abb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:01:46.597100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:46.650084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 600/1194] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:01:46 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:01:46.928932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e56ca-3624-45f0-8ca0-27f884e5c210 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:46.933872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:47.038716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35e56ca-3624-45f0-8ca0-27f884e5c210 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:47.073728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:47.074378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:47.074651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:47.074923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:47.085938 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:01:47.090698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:01:47.092177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:01:47.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 595/1195] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:01:46 2022] GET /identity/v3/roles/admin => generated 83 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:01:47.098059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd3253e-8ab3-4bbd-8869-06660c686abb None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:47.202372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddd3253e-8ab3-4bbd-8869-06660c686abb None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:47.241663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:47.242289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:47.242564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:47.242833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:47.243494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:47.243793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:47.245166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:47.255379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 601/1196] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:01:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:01:47.265676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:47.335059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:47.387585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c87391-c002-43a8-8d8d-2b6f8e9a63ae None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:47.422972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:47.423580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:47.423851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:47.424120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] PATH_INFO: `/v3/users/b07e7fa4fbae41bcb4856638c0dbcdce` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:47.425362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] RBAC: Authorizing `identity:get_user(user_id=b07e7fa4fbae41bcb4856638c0dbcdce)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:47.427187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:47.428912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 596/1197] 10.222.0.61 () {56 vars in 1177 bytes} [Thu Sep 29 02:01:47 2022] GET /identity/v3/users/b07e7fa4fbae41bcb4856638c0dbcdce => generated 311 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:47.438600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:47.463453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:47.555094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cf5735-5883-486b-b66b-30e3d343a223 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:47.588183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193b3284-a358-4ed8-b866-984333921882 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:47.588790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193b3284-a358-4ed8-b866-984333921882 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:47.589024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193b3284-a358-4ed8-b866-984333921882 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:47.589288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193b3284-a358-4ed8-b866-984333921882 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:47.664842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193b3284-a358-4ed8-b866-984333921882 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:47.666308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193b3284-a358-4ed8-b866-984333921882 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:47.681601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:47.767747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 602/1198] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:01:47 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 330 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:01:47.777223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:47.884765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a52716-e460-4b1e-a90d-55e412d954e2 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:47.914301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:47.922662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73bfe01c-ece2-48c8-b995-4ee723501654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:47.923297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bfe01c-ece2-48c8-b995-4ee723501654 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:47.923574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bfe01c-ece2-48c8-b995-4ee723501654 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:47.923844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73bfe01c-ece2-48c8-b995-4ee723501654 None admin] PATH_INFO: `/v3/projects/7e92e97b91e04df18fece2fc5ce713e8` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:47.925082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bfe01c-ece2-48c8-b995-4ee723501654 None admin] RBAC: Authorizing `identity:get_project(project_id=7e92e97b91e04df18fece2fc5ce713e8)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:47.926916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73bfe01c-ece2-48c8-b995-4ee723501654 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:47.929056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 597/1199] 10.222.0.61 () {56 vars in 1186 bytes} [Thu Sep 29 02:01:47 2022] GET /identity/v3/projects/7e92e97b91e04df18fece2fc5ce713e8 => generated 361 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:47.935382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193b3284-a358-4ed8-b866-984333921882 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:01:48.037651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193b3284-a358-4ed8-b866-984333921882 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:01:48.051716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:48.085253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbf8b2cb-dec2-4565-a25f-58a802786be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:01:48.085846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf8b2cb-dec2-4565-a25f-58a802786be0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:01:48.086153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf8b2cb-dec2-4565-a25f-58a802786be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:01:48.086423 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf8b2cb-dec2-4565-a25f-58a802786be0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:01:48.087143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf8b2cb-dec2-4565-a25f-58a802786be0 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=7e92e97b91e04df18fece2fc5ce713e8, user.id=b07e7fa4fbae41bcb4856638c0dbcdce) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:01:48.087452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf8b2cb-dec2-4565-a25f-58a802786be0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:01:48.088887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbf8b2cb-dec2-4565-a25f-58a802786be0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:01:48.095435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 603/1200] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:01:47 2022] GET /identity/v3/role_assignments?user.id=b07e7fa4fbae41bcb4856638c0dbcdce&scope.project.id=7e92e97b91e04df18fece2fc5ce713e8&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:01:57.920071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:01:58.093364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=11 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:44.289062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d7a47a-339a-4d0d-8c78-ea3c6847856e None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:44.289758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d7a47a-339a-4d0d-8c78-ea3c6847856e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:44.289758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d7a47a-339a-4d0d-8c78-ea3c6847856e None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:44.291415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 598/1201] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:02:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:44.297499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e92e2-7eff-4327-9be4-52a6d0cbae96 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:44.298139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e92e2-7eff-4327-9be4-52a6d0cbae96 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:44.298242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5e92e2-7eff-4327-9be4-52a6d0cbae96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:44.319952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bd5e92e2-7eff-4327-9be4-52a6d0cbae96 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:44.321247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bd5e92e2-7eff-4327-9be4-52a6d0cbae96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:44.406798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 604/1202] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:44.677578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d7a47a-339a-4d0d-8c78-ea3c6847856e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:44.812715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d7a47a-339a-4d0d-8c78-ea3c6847856e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:44.859387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:44.860113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:44.860404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:44.860651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:44.872191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:44.873779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:44.890620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:44.891715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:44.892663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:44.892663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:44.892663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:02:44.892663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 599/1203] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:44.898472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e92e2-7eff-4327-9be4-52a6d0cbae96 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:45.021913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5e92e2-7eff-4327-9be4-52a6d0cbae96 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:45.069593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:45.077064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65252f53-fe7b-4a9a-985a-4b4f9f930fc3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:45.077934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65252f53-fe7b-4a9a-985a-4b4f9f930fc3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:45.078323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65252f53-fe7b-4a9a-985a-4b4f9f930fc3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:45.078599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65252f53-fe7b-4a9a-985a-4b4f9f930fc3 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:45.079279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65252f53-fe7b-4a9a-985a-4b4f9f930fc3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:45.079605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65252f53-fe7b-4a9a-985a-4b4f9f930fc3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:45.081136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65252f53-fe7b-4a9a-985a-4b4f9f930fc3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:45.096860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 605/1204] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:02:45.103866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:45.179527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:45.239496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc146b71-afb8-416e-a1c7-55059519e16e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:45.278228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53686a7c-426b-43b2-b8ed-3c8ed67d6a2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:45.278228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53686a7c-426b-43b2-b8ed-3c8ed67d6a2d None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:45.278228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53686a7c-426b-43b2-b8ed-3c8ed67d6a2d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:45.278228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53686a7c-426b-43b2-b8ed-3c8ed67d6a2d None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:45.279419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53686a7c-426b-43b2-b8ed-3c8ed67d6a2d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:45.280950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53686a7c-426b-43b2-b8ed-3c8ed67d6a2d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:45.314316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 600/1205] 10.222.0.61 () {60 vars in 1132 bytes} [Thu Sep 29 02:02:45 2022] POST /identity/v3/users => generated 251 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:02:47.704671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e899b-27bf-453e-8ec6-1132c2e6dcd1 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:47.705251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e899b-27bf-453e-8ec6-1132c2e6dcd1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:47.705625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775e899b-27bf-453e-8ec6-1132c2e6dcd1 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:47.707376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 606/1206] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:02:47 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:47.716230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f07b1a-7732-41dd-879e-57badb6b7250 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:47.716907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f07b1a-7732-41dd-879e-57badb6b7250 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:47.716907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f07b1a-7732-41dd-879e-57badb6b7250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:47.737229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b3f07b1a-7732-41dd-879e-57badb6b7250 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:47.738470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b3f07b1a-7732-41dd-879e-57badb6b7250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:47.817400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 601/1207] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:48.108836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e899b-27bf-453e-8ec6-1132c2e6dcd1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:48.235454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775e899b-27bf-453e-8ec6-1132c2e6dcd1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:48.280047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:48.280734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:48.281049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:48.281358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:48.293029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:02:48.297768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:48.298853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:48.300686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 607/1208] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 02:02:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:48.306449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f07b1a-7732-41dd-879e-57badb6b7250 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:48.351765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:48.419763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f07b1a-7732-41dd-879e-57badb6b7250 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:48.457507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:48.458270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:48.458596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:48.458822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:48.459528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:48.459839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:48.461257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:48.475182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 602/1209] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 02:02:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:48.476647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:48.537724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:48.598119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86765613-2dab-4251-af2d-3d481da07c4a None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:48.637960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:48.639308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:48.639616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:48.639914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:48.652214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:48.653888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:48.669951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:48.670993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:48.672082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:48.672082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:48.672082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:48.675792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 608/1210] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:48.678285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:48.805962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:48.812002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d38c52-a252-4c8c-bfcb-2232f8078d5b None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:48.852322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:48.853387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:48.853387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:48.853687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:48.854312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:48.854670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:48.856108 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:48.868943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 603/1211] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:48.874771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:48.985395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b126ec-8b36-46ce-a01e-12bdf904bba4 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:48.986943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:49.020768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:49.021428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:49.021698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:49.021961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:49.030168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:49.031606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:02:49.044098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:49.045176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:49.046637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 609/1212] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:48 2022] GET /identity/v3/users/placement => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:02:49.051291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:49.169904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe46d5-f86a-4988-9627-fe63d80de729 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:49.214507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:49.215365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:49.215720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:49.216023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:49.216845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:49.217144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:49.221170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:49.241200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 604/1213] 10.222.0.61 () {56 vars in 1143 bytes} [Thu Sep 29 02:02:49 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:49.247281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:49.361489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66f466c-4bd7-4195-8947-c1bbd1fcbbc2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:49.407034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:49.407640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:49.407939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:49.408282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:49.440872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:49.486312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:49.488297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:49.602474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 610/1214] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:02:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 356 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:02:49.608655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:49.630361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:49.717341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bb55460-dd9a-4249-9abd-ffee7b369de1 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:49.755783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:49.756515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:49.756800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:49.757069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:49.768550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:49.770338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:49.786230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:49.787651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:49.788777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:49.788777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:49.788777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:02:49.788777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 605/1215] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:49.795863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:49.867178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:49.918351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9935217-ef1f-4abe-ad96-31aee1206d17 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:49.954573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:49.955184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:49.955463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:49.955734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:49.956376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:49.956675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:49.958072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:49.970649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 611/1216] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:49.976949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:50.050620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:50.087526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fd94df6-db48-46ae-b164-51497e75fdcd None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:50.124051 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:50.124675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:50.124952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:50.125227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:50.133036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:50.134847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:02:50.146735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:50.147967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:02:50.148941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 606/1217] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:02:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:02:50.154573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:50.266599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d42aa-699b-4e44-b7e1-1865527abf40 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:50.291572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:50.305465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf77dea3-f7f3-4ff0-b5e6-9354941cebf0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:50.306233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77dea3-f7f3-4ff0-b5e6-9354941cebf0 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:50.306616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77dea3-f7f3-4ff0-b5e6-9354941cebf0 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:50.306983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf77dea3-f7f3-4ff0-b5e6-9354941cebf0 None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:50.307867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf77dea3-f7f3-4ff0-b5e6-9354941cebf0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:50.308212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf77dea3-f7f3-4ff0-b5e6-9354941cebf0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:50.309805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf77dea3-f7f3-4ff0-b5e6-9354941cebf0 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:50.322881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 612/1218] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:02:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:50.329065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:50.441909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ead7d6-8217-4ffb-93c5-c9163c29af30 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:50.472393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:50.479304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b51ce94-8d02-4598-9345-0501b0344118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:50.479959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b51ce94-8d02-4598-9345-0501b0344118 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:50.480462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b51ce94-8d02-4598-9345-0501b0344118 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:50.480762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b51ce94-8d02-4598-9345-0501b0344118 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:50.481573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b51ce94-8d02-4598-9345-0501b0344118 None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=8083af87f4f742be8b769d8b0bb7ca6f) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:50.481865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b51ce94-8d02-4598-9345-0501b0344118 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:50.483544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b51ce94-8d02-4598-9345-0501b0344118 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:50.494041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 607/1219] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:02:50 2022] GET /identity/v3/role_assignments?user.id=8083af87f4f742be8b769d8b0bb7ca6f&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 259 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:52.699936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e30dc97-cacf-4e1a-9bad-8fbd3b260816 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:52.700536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e30dc97-cacf-4e1a-9bad-8fbd3b260816 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:52.700536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e30dc97-cacf-4e1a-9bad-8fbd3b260816 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:52.701850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 613/1220] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:02:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:52.707023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d202d2-cb22-46cb-be29-72301a3a0ab6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:52.707473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d202d2-cb22-46cb-be29-72301a3a0ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:52.707579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d202d2-cb22-46cb-be29-72301a3a0ab6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:52.726445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-67d202d2-cb22-46cb-be29-72301a3a0ab6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:52.728216 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-67d202d2-cb22-46cb-be29-72301a3a0ab6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:52.798068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 608/1221] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:53.083849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e30dc97-cacf-4e1a-9bad-8fbd3b260816 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:53.203810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e30dc97-cacf-4e1a-9bad-8fbd3b260816 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:53.238177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:53.238795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:53.239107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:53.239339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:53.249518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:02:53.254342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:53.255590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:53.256494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 614/1222] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 02:02:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:53.262185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d202d2-cb22-46cb-be29-72301a3a0ab6 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:53.367485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d202d2-cb22-46cb-be29-72301a3a0ab6 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:53.396227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:53.408830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:53.409574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:53.409884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:53.410292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:53.411166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:53.411488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:53.413205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:53.423114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 609/1223] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 02:02:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:53.429619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:53.476844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:53.537238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf06077-4070-45be-b07d-df9b41327549 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:53.575764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:53.576670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:53.577089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:53.577464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:53.591407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:53.592973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:53.608998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:53.610283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:53.611398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:53.611398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:53.611398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:53.611398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 615/1224] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:02:53.619257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:53.740896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e272e2ff-1305-46ad-8541-a4b23d43b4a7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:53.779769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:53.780657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:53.780970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:53.781244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:53.782072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:53.782373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:53.784004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:53.800316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 610/1225] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:02:53.807399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:53.905314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:53.913711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77806c6d-0933-4ce0-a5f9-47717f9793b9 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:53.951532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:53.952170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:53.954061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:53.954061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:53.960748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:53.962571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:02:53.974521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:53.976117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:53.977959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 616/1226] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:53.983333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:54.059071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:54.122574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880d4e3b-8823-4bef-a145-13798443e761 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:54.160402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:54.160940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:54.161220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:54.161493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:54.162251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:54.162556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:54.163996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:54.183377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 611/1227] 10.222.0.61 () {56 vars in 1143 bytes} [Thu Sep 29 02:02:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:54.189454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:54.303647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-023b36ff-829f-4beb-a9c7-ba63792f0f7b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:54.340924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:54.341554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:54.341826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:54.342122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:54.354270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:54.355761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:54.370585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:54.371735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:54.372906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:54.372906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:54.372906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:54.372906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 617/1228] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:54.377679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:54.485793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d12203e0-1b1f-42cd-9ea8-0c5bf970cde3 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:54.488647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:54.524067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:54.525000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:54.525275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:54.525575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:54.526325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:54.526657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:54.528171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:54.541279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 612/1229] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:54.550172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:54.561476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:54.660665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9239e069-b0e7-4efe-8a6d-1f0d1290ad2e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:54.699153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:54.699826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:54.700139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:54.700413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:54.708922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:54.710440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:02:54.723201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:54.724419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:54.725716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 618/1230] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:02:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:54.730613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:54.835625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8e0bc3-94d3-40cb-b8fa-17524526f82c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:54.873289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-142ce2b0-30ee-4646-bd2d-ead7e0fcccf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:54.873906 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ce2b0-30ee-4646-bd2d-ead7e0fcccf9 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:54.874209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ce2b0-30ee-4646-bd2d-ead7e0fcccf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:54.874478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-142ce2b0-30ee-4646-bd2d-ead7e0fcccf9 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:54.875171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ce2b0-30ee-4646-bd2d-ead7e0fcccf9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:54.875486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ce2b0-30ee-4646-bd2d-ead7e0fcccf9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:54.876929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-142ce2b0-30ee-4646-bd2d-ead7e0fcccf9 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:54.889532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 613/1231] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:02:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:54.895887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:54.963941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:55.003198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c456f3d0-06a7-4597-8858-736ebe5f06d2 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:55.040468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9115356-89cb-43b8-bfb5-718d7c3f4c84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:55.041243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9115356-89cb-43b8-bfb5-718d7c3f4c84 None admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:55.041580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9115356-89cb-43b8-bfb5-718d7c3f4c84 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:55.041960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9115356-89cb-43b8-bfb5-718d7c3f4c84 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/8083af87f4f742be8b769d8b0bb7ca6f/roles/5c49b35bed6a4d3290bab54c9fa73b9e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:55.057374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9115356-89cb-43b8-bfb5-718d7c3f4c84 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=8083af87f4f742be8b769d8b0bb7ca6f, role_id=5c49b35bed6a4d3290bab54c9fa73b9e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:55.058848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9115356-89cb-43b8-bfb5-718d7c3f4c84 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:55.073193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 619/1232] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:02:54 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/8083af87f4f742be8b769d8b0bb7ca6f/roles/5c49b35bed6a4d3290bab54c9fa73b9e => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:02:57.273320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497c4dd6-d3fe-4000-8cab-1790211f4efb None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:57.273864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497c4dd6-d3fe-4000-8cab-1790211f4efb None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:57.273864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497c4dd6-d3fe-4000-8cab-1790211f4efb None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:57.275411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 614/1233] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:02:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:02:57.280851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb730b-567e-4ef1-8317-b40cffa40262 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:57.281283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb730b-567e-4ef1-8317-b40cffa40262 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:57.281538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4acb730b-567e-4ef1-8317-b40cffa40262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:57.301479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4acb730b-567e-4ef1-8317-b40cffa40262 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:02:57.303112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4acb730b-567e-4ef1-8317-b40cffa40262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:02:57.377487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 620/1234] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:02:57 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:02:57.663991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497c4dd6-d3fe-4000-8cab-1790211f4efb None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:57.692373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:57.777219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497c4dd6-d3fe-4000-8cab-1790211f4efb None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:57.815912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:57.816604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:57.816904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:57.817200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:57.828693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:02:57.833921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:57.835200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:02:57.836063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 615/1235] 10.222.0.61 () {56 vars in 1102 bytes} [Thu Sep 29 02:02:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:57.841870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acb730b-567e-4ef1-8317-b40cffa40262 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:57.892535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:57.949597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4acb730b-567e-4ef1-8317-b40cffa40262 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:57.987648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:57.987911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:57.988192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:57.988498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:57.989207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] RBAC: Adding query filter params (name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:57.989562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:57.990971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:58.005012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 621/1236] 10.222.0.61 () {56 vars in 1103 bytes} [Thu Sep 29 02:02:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:58.006395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:58.105001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:58.125486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7005ce-590a-460d-84e7-b3ad2a0e64d0 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:58.164230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:58.165094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:58.165552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:58.165823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:58.176834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:58.178635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:58.193774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:58.195032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:58.196202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:58.196202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:58.196202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:02:58.196202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 616/1237] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:58.201459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:58.312599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74994090-b449-49c5-98c0-3fa09b225cce None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:58.322583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:58.348723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:58.349529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:58.349859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:58.350194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:58.350921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:58.351268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:58.353353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:58.368012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 622/1238] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:02:58.373797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:58.489716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4beb9d45-c8b3-4845-97cc-8b7349bcbe1c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:58.535068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:58.535829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:58.536109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:58.536384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:58.539545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:58.549592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:58.549592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:02:58.562568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:58.563729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:02:58.564903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:02:58.566811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 617/1239] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:58.573100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:58.690152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4658e56b-6a86-4c45-8520-c21700eea23f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:58.726688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:58.727916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:58.728249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:58.728519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:58.729263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:58.729551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:58.731031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:58.749003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 623/1240] 10.222.0.61 () {56 vars in 1143 bytes} [Thu Sep 29 02:02:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:58.755229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:58.866148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01eed45-493b-48e5-b023-b9684c244441 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:58.905770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:58.906421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:58.906916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:58.907457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:58.973592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:58.980419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:58.981891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:59.084031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 618/1241] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:02:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 29 02:02:59.090096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:59.119233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:59.199090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81313ead-54e4-4a53-ba5e-000ac691dd4f None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:59.237457 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:59.238024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:59.238300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:59.238565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:59.248735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:59.250209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:02:59.265505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:59.266646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:02:59.267555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:02:59.267555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:02:59.267555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:59.267555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 624/1242] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:02:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:59.272646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:59.376641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:59.383028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca28f45a-e299-4f63-b0a7-8e3cbb7343bc None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:59.418019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:59.418653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:59.418924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:59.419196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:59.419829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:59.420123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:59.421499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:59.433738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 619/1243] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:02:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:02:59.439544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:59.525243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:02:59.553748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4918a589-dc1e-4369-837a-79ef9c5232d7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:59.589007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:59.589645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:59.589888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:59.590189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:59.596982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:59.598475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:02:59.611671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:02:59.612683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:02:59.613591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 625/1244] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:02:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:02:59.619520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:59.735440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007991b5-6f1e-4d6b-897f-5d006d6ff702 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:59.773190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d330654-9453-4330-96cb-dd1cd975beb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:59.773800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d330654-9453-4330-96cb-dd1cd975beb4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:59.774515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d330654-9453-4330-96cb-dd1cd975beb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:59.774515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d330654-9453-4330-96cb-dd1cd975beb4 None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:59.775103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d330654-9453-4330-96cb-dd1cd975beb4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:59.775457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d330654-9453-4330-96cb-dd1cd975beb4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:59.776849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d330654-9453-4330-96cb-dd1cd975beb4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:59.790064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 620/1245] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:02:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:02:59.796226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:02:59.907274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb191d81-7300-4e7a-9afa-1d8cdc09e648 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:02:59.950335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9fe58a-3219-4df3-bb26-81cd6402cb3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:02:59.950944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9fe58a-3219-4df3-bb26-81cd6402cb3a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:02:59.951216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9fe58a-3219-4df3-bb26-81cd6402cb3a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:02:59.951511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9fe58a-3219-4df3-bb26-81cd6402cb3a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:02:59.952262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9fe58a-3219-4df3-bb26-81cd6402cb3a None admin] RBAC: Adding query filter params (role.id=5c49b35bed6a4d3290bab54c9fa73b9e, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=8083af87f4f742be8b769d8b0bb7ca6f) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:02:59.952565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9fe58a-3219-4df3-bb26-81cd6402cb3a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:02:59.954012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9fe58a-3219-4df3-bb26-81cd6402cb3a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:02:59.966189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 626/1246] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:02:59 2022] GET /identity/v3/role_assignments?user.id=8083af87f4f742be8b769d8b0bb7ca6f&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=5c49b35bed6a4d3290bab54c9fa73b9e => generated 608 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:02.422218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8502f5-2135-4edd-a52e-610c9d185e4e None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:02.422804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8502f5-2135-4edd-a52e-610c9d185e4e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:02.422804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8502f5-2135-4edd-a52e-610c9d185e4e None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:02.424096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 621/1247] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:03:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:02.430550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0733758c-6269-41fa-9b01-c33ced4917ce None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:02.431479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0733758c-6269-41fa-9b01-c33ced4917ce None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:02.431768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0733758c-6269-41fa-9b01-c33ced4917ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:02.438444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:02.453155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0733758c-6269-41fa-9b01-c33ced4917ce None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:02.454816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0733758c-6269-41fa-9b01-c33ced4917ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:02.541209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 627/1248] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:02 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:03:02.805194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8502f5-2135-4edd-a52e-610c9d185e4e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:02.827104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:02.927334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8502f5-2135-4edd-a52e-610c9d185e4e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:02.965626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:02.966336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:02.966618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:02.966986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:02.979433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:03:02.984260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:02.985474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:03:02.986728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:02.990268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 622/1249] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:03:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:02.991642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0733758c-6269-41fa-9b01-c33ced4917ce None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:03.092797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0733758c-6269-41fa-9b01-c33ced4917ce None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:03.127397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:03.133188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:03.134313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:03.134448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:03.134629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:03.135292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:03.135587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:03.136951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:03.146122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 628/1250] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:03:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:03.152151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:03.245511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:03.272179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17943c0-05d6-46ad-887a-aaf390edfc51 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:03.309527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:03.310206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:03.310478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:03.310746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:03.321452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:03.323063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:03:03.338050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:03.339139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:03:03.340045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:03:03.340045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:03.340045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:03.340045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 623/1251] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:03.345066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:03.451304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9020fe4f-dde1-45b3-8729-1f027e14aa8b None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:03.489136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:03.489908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:03.490219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:03.490536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:03.491304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:03.491687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:03.493226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:03.507509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 629/1252] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:03:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:03.513364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:03.625358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b26d1322-0aef-4810-a2d3-0a135b54329c None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:03.653122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:03.661425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:03.662072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:03.662344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:03.662614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:03.670975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:03.672411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:03:03.684042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:03.690180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:03.691232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 624/1253] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:03 2022] GET /identity/v3/users/placement => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:03.691838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:03.752753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:03.813797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa651c5-694b-4873-a3b1-595d8109183c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:03.853735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:03.854555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:03.854842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:03.855126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:03.855992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:03.856382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:03.857851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:03.877316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 630/1254] 10.222.0.61 () {56 vars in 1143 bytes} [Thu Sep 29 02:03:03 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:03.883449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:04.005189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3709a3e-4167-47c2-9c36-7fc7d6fc87a7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:04.047149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:04.047765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:04.048037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:04.048313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:04.123247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:04.141130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:04.143194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:04.299378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 625/1255] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:03:03 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:03:04.306669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:04.446949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8289775-0bbc-498a-a0e4-deeaf8a9637c None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:04.463774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:04.487448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:04.488175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:04.488448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:04.488718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:04.501353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:04.502981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:03:04.519497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:04.520729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:03:04.521770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:03:04.521770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:04.521770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:04.526229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 631/1256] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:04.528155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:04.639227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:04.652900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b084d3a-533d-4682-b52a-531547c1a2f7 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:04.696407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:04.697198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:04.697343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:04.697572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:04.698228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:04.698527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:04.699929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:04.713628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 626/1257] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:03:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:04.719807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:04.853070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fac542c-f3e7-4fb8-85c2-abb0e1976829 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:04.892755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:04.893492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:04.893870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:04.894296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:04.902319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:04.904224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:03:04.919483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:04.920562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:04.921451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 632/1258] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:03:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:04.928144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:05.045061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a57e489-25a7-481d-a49d-369942c62a20 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:05.089535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41cadcbf-f694-4606-991b-6fcd912aacaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:05.090199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cadcbf-f694-4606-991b-6fcd912aacaf None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:05.090487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cadcbf-f694-4606-991b-6fcd912aacaf None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:05.090737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41cadcbf-f694-4606-991b-6fcd912aacaf None admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:05.091428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cadcbf-f694-4606-991b-6fcd912aacaf None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:05.091723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cadcbf-f694-4606-991b-6fcd912aacaf None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:05.093185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41cadcbf-f694-4606-991b-6fcd912aacaf None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:05.106875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 627/1259] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:03:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:05.113333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:05.144623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:05.240781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f0edccf-ae2b-45c2-8034-85ce1e7f99e7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:05.280587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba0bce5-a73a-471c-82a9-77e2ece9c98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:05.281191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba0bce5-a73a-471c-82a9-77e2ece9c98f None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:05.281462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba0bce5-a73a-471c-82a9-77e2ece9c98f None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:05.281725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba0bce5-a73a-471c-82a9-77e2ece9c98f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:05.282484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba0bce5-a73a-471c-82a9-77e2ece9c98f None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=8083af87f4f742be8b769d8b0bb7ca6f) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:05.282786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba0bce5-a73a-471c-82a9-77e2ece9c98f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:05.284257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba0bce5-a73a-471c-82a9-77e2ece9c98f None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:05.290844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 633/1260] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:03:05 2022] GET /identity/v3/role_assignments?user.id=8083af87f4f742be8b769d8b0bb7ca6f&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:07.409404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d1d9bd-08a8-417d-99b9-57e54b825bb8 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:07.410021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d1d9bd-08a8-417d-99b9-57e54b825bb8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:07.410021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d1d9bd-08a8-417d-99b9-57e54b825bb8 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:07.411310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 628/1261] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:03:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:07.417672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59822db9-f738-4a0d-b37b-e37263e6cf45 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:07.418420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59822db9-f738-4a0d-b37b-e37263e6cf45 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:07.418580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59822db9-f738-4a0d-b37b-e37263e6cf45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:07.443951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-59822db9-f738-4a0d-b37b-e37263e6cf45 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:07.445154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-59822db9-f738-4a0d-b37b-e37263e6cf45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:07.523334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 634/1262] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:07 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:03:07.787908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d1d9bd-08a8-417d-99b9-57e54b825bb8 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:07.800564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:07.901971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d1d9bd-08a8-417d-99b9-57e54b825bb8 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:07.937265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:07.937888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:07.938197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:07.938463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:07.949894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:03:07.954638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:07.955716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:07.956711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 629/1263] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:03:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:07.963655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59822db9-f738-4a0d-b37b-e37263e6cf45 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:07.988226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:08.075882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59822db9-f738-4a0d-b37b-e37263e6cf45 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:08.115208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:08.115807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:08.116074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:08.116341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:08.116997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] RBAC: Adding query filter params (name=admin) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:08.117296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:08.119013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:08.127643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 635/1264] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:03:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:08.136891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:08.222102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:08.258887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea9d195-00ba-4360-881a-cc403dd79a27 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:08.293600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:08.294244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:08.294511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:08.294770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:08.305275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:08.306862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:03:08.321200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:08.322346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:03:08.323377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:03:08.323377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:08.323377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:08.323377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 630/1265] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:08.328781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:08.413144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:08.441309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-212b8878-07ed-4b6e-afd9-ef8db5d9d8cd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:08.480035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:08.480634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:08.480908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:08.481178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:08.481812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:08.482145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:08.483541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:08.499182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 636/1266] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:03:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:08.508844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:08.624809 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df69f895-4499-4da0-8a2e-7b1b0228a933 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:08.644872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:08.660706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:08.661328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:08.661601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:08.661866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] PATH_INFO: `/v3/users/placement` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:08.669528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:08.671030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:03:08.682575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:08.683664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:08.684610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 631/1267] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:08 2022] GET /identity/v3/users/placement => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:08.689842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:08.799196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4747e4ed-f98b-48e7-975a-89552e1cfedd None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:08.835188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:08.835784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:08.836059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:08.836327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:08.837158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:08.837473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:08.838999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:08.847765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:08.857171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 637/1268] 10.222.0.61 () {56 vars in 1143 bytes} [Thu Sep 29 02:03:08 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:08.863308 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:08.980574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99e114a-4766-489d-8c8b-9744c2960f3a None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:09.023116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:09.023743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:09.024016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:09.024411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:09.041621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:09.043962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:03:09.063400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:09.064512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:03:09.065765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:03:09.065765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:09.065765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:09.065765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 632/1269] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:09.070552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:09.177417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02274b72-9e02-4ee4-8c14-e66e562f9ec5 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:09.218815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:09.219452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:09.219728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:09.219998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:09.220659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:09.220961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:09.222432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:09.235527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 638/1270] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:03:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:09.241357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:09.276060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:09.345543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84331068-0f91-4140-a022-6c5e427ee4d5 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:09.379003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:09.379610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:09.379880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:09.380144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:09.387003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:09.388438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:03:09.399749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:09.400666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:09.401681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 633/1271] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:03:09 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:09.406996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:09.469087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:09.516600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-734b91c4-5694-43e5-8a3a-8fa4ff3ce87d None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:09.554154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d5c7bdb-ffa3-418f-871e-8e6b1841129d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:09.554774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c7bdb-ffa3-418f-871e-8e6b1841129d None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:09.555038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c7bdb-ffa3-418f-871e-8e6b1841129d None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:09.555313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d5c7bdb-ffa3-418f-871e-8e6b1841129d None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:09.556079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c7bdb-ffa3-418f-871e-8e6b1841129d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:09.556441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c7bdb-ffa3-418f-871e-8e6b1841129d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:09.557884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d5c7bdb-ffa3-418f-871e-8e6b1841129d None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:09.570222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 639/1272] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:03:09 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:09.576592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:09.681939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:09.692150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a64a9c2-1d4e-453a-9589-b9e996c6f0c4 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:09.727688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-114654bc-7c11-447a-a9aa-935db9caa435 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:09.728332 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114654bc-7c11-447a-a9aa-935db9caa435 None admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:09.728608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114654bc-7c11-447a-a9aa-935db9caa435 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:09.728879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-114654bc-7c11-447a-a9aa-935db9caa435 None admin] PATH_INFO: `/v3/projects/ae7c64039839431980d6f61231270abc/users/8083af87f4f742be8b769d8b0bb7ca6f/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:09.730875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114654bc-7c11-447a-a9aa-935db9caa435 None admin] RBAC: Authorizing `identity:create_grant(project_id=ae7c64039839431980d6f61231270abc, user_id=8083af87f4f742be8b769d8b0bb7ca6f, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:09.732367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-114654bc-7c11-447a-a9aa-935db9caa435 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:09.747668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 634/1273] 10.222.0.61 () {58 vars in 1439 bytes} [Thu Sep 29 02:03:09 2022] PUT /identity/v3/projects/ae7c64039839431980d6f61231270abc/users/8083af87f4f742be8b769d8b0bb7ca6f/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:03:11.849844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5716f3-931d-408d-b0af-29994e44efa0 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:11.850774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5716f3-931d-408d-b0af-29994e44efa0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:11.850774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5716f3-931d-408d-b0af-29994e44efa0 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:11.853874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 640/1274] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:03:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:11.860192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4813c53c-e439-4b8b-8c2f-7b6ae1dbb10f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:11.860909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4813c53c-e439-4b8b-8c2f-7b6ae1dbb10f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:11.861092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4813c53c-e439-4b8b-8c2f-7b6ae1dbb10f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:11.882875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-4813c53c-e439-4b8b-8c2f-7b6ae1dbb10f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:11.884605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4813c53c-e439-4b8b-8c2f-7b6ae1dbb10f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:11.954014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 635/1275] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:03:12.234223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5716f3-931d-408d-b0af-29994e44efa0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:12.352583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5716f3-931d-408d-b0af-29994e44efa0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:12.370467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:12.391050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:12.391676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:12.391945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:12.392198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:12.403557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 29 02:03:12.408297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:12.409309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:12.410243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 641/1276] 10.222.0.61 () {56 vars in 1096 bytes} [Thu Sep 29 02:03:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:12.416448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4813c53c-e439-4b8b-8c2f-7b6ae1dbb10f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:12.505804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:12.529579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4813c53c-e439-4b8b-8c2f-7b6ae1dbb10f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:12.565439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:12.566041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:12.566319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:12.566587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:12.567246 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] RBAC: Adding query filter params (name=admin) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:12.567545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:12.568920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:12.577153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 636/1277] 10.222.0.61 () {56 vars in 1099 bytes} [Thu Sep 29 02:03:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:03:12.585924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:12.689414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cfa72-3109-41d6-a8c6-43486165eec3 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:12.725709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:12.726344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:12.726614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:12.726881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:12.738032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:12.739476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:03:12.754349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:12.755511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:03:12.756700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:03:12.756700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:12.756700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:12.756700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 642/1278] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:12.766690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:12.879732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987859df-1ee4-463e-94bb-fd6e8db792af None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:12.917239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:12.917826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:12.918126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:12.918393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:12.919022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:12.919313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:12.920681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:12.933542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 637/1279] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:03:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:12.942737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:12.977416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:13.067727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0319037-b5a8-4d5b-869e-8dc2c15f8f19 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:13.103587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:13.104200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:13.104483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:13.104766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:13.112456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:13.113887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 29 02:03:13.125171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:13.126197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:13.127268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 643/1280] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:12 2022] GET /identity/v3/users/placement => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:13.135398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:13.139436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:13.239555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ee3b8ca-3db9-406e-9e08-f1b1195362ee None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:13.275715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:13.276317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:13.276591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:13.276859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:13.277586 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:13.277885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:13.279349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:13.301145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 638/1281] 10.222.0.61 () {56 vars in 1143 bytes} [Thu Sep 29 02:03:13 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:03:13.309542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:13.415650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:13.426483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1fd952-08ac-4f85-b121-f5cae2b7dfa8 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:13.461821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:13.462447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:13.462693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:13.462971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:13.536838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:13.538311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:13.632009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:13.644191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 644/1282] 10.222.0.61 () {58 vars in 1282 bytes} [Thu Sep 29 02:03:13 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 29 02:03:13.651434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:13.733646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:13.782334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d0632-cecd-498d-9629-c5d911a5c79e None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:13.817337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:13.817947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:13.818244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:13.818505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:13.828634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:13.830061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 29 02:03:13.844590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:13.845631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 29 02:03:13.846721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 29 02:03:13.846721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 29 02:03:13.846721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:13.846721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 639/1283] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:03:13 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:03:13.854037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:13.954030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7a0a-cc33-418f-b189-72c00c703419 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:13.995930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:13.996447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:13.996716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:13.996984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:13.997611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:13.997910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:13.999280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:14.011418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 645/1284] 10.222.0.61 () {56 vars in 1109 bytes} [Thu Sep 29 02:03:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:03:14.019812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:14.127435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a59779c4-db30-4cee-adb6-36d0dfbc2eb6 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:14.140927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:14.162164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:14.162788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:14.163065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:14.163334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] PATH_INFO: `/v3/projects/service` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:14.170193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:14.171645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 29 02:03:14.183526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:14.189368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: ERROR keystone.server.flask.application Sep 29 02:03:14.190492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 640/1285] 10.222.0.61 () {56 vars in 1111 bytes} [Thu Sep 29 02:03:14 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 29 02:03:14.191129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:14.243557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:14.294302 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-683e0ba3-dd80-4a7e-92c1-84d2a41a49f7 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:14.334831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b3ee40-ef06-40f0-bf60-9df7ebbeb59a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:14.335435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b3ee40-ef06-40f0-bf60-9df7ebbeb59a None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:14.335709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b3ee40-ef06-40f0-bf60-9df7ebbeb59a None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:14.335983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b3ee40-ef06-40f0-bf60-9df7ebbeb59a None admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:14.336682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b3ee40-ef06-40f0-bf60-9df7ebbeb59a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:14.337059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b3ee40-ef06-40f0-bf60-9df7ebbeb59a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:14.338695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b3ee40-ef06-40f0-bf60-9df7ebbeb59a None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:14.350843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 646/1286] 10.222.0.61 () {56 vars in 1148 bytes} [Thu Sep 29 02:03:14 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:14.357234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:14.468321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f73fe3c-5200-4890-95c0-6c69ed27c65d None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:14.503526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c7b879-040e-4e7a-88fd-5de9dbb84560 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:14.504116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7b879-040e-4e7a-88fd-5de9dbb84560 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:14.504390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7b879-040e-4e7a-88fd-5de9dbb84560 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:14.504657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c7b879-040e-4e7a-88fd-5de9dbb84560 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:14.505370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c7b879-040e-4e7a-88fd-5de9dbb84560 None admin] RBAC: Adding query filter params (role.id=69aa90da9c204a0fbb53f7289b184185, scope.project.id=ae7c64039839431980d6f61231270abc, user.id=8083af87f4f742be8b769d8b0bb7ca6f) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:14.505676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c7b879-040e-4e7a-88fd-5de9dbb84560 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:14.507143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c7b879-040e-4e7a-88fd-5de9dbb84560 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:14.513437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 641/1287] 10.222.0.61 () {56 vars in 1374 bytes} [Thu Sep 29 02:03:14 2022] GET /identity/v3/role_assignments?user.id=8083af87f4f742be8b769d8b0bb7ca6f&scope.project.id=ae7c64039839431980d6f61231270abc&role.id=69aa90da9c204a0fbb53f7289b184185 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:16.665067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc7f214-3ae6-4294-a927-9cd6753466ad None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:16.665656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc7f214-3ae6-4294-a927-9cd6753466ad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:16.665656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc7f214-3ae6-4294-a927-9cd6753466ad None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:16.667006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 647/1288] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:03:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:16.673363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fe0cb4-2feb-4f8a-8673-cc604f94c2c3 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:16.674074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fe0cb4-2feb-4f8a-8673-cc604f94c2c3 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:16.674215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fe0cb4-2feb-4f8a-8673-cc604f94c2c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:16.697387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-50fe0cb4-2feb-4f8a-8673-cc604f94c2c3 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:16.698597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-50fe0cb4-2feb-4f8a-8673-cc604f94c2c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:16.732003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:16.772396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 642/1289] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:16 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:03:17.037442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc7f214-3ae6-4294-a927-9cd6753466ad None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:17.157966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc7f214-3ae6-4294-a927-9cd6753466ad None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:17.170120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:17.199763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:17.200384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:17.200663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:17.200930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] PATH_INFO: `/v3/services/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:17.201507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:17.202893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:17.213433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:17.214933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:17.216277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 648/1290] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:03:17 2022] GET /identity/v3/services/placement => generated 90 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:17.220898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fe0cb4-2feb-4f8a-8673-cc604f94c2c3 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:17.318510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fe0cb4-2feb-4f8a-8673-cc604f94c2c3 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:17.353955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a756e71-8b34-42d1-ac50-f74d104dd36c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:17.354582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a756e71-8b34-42d1-ac50-f74d104dd36c None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:17.354855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a756e71-8b34-42d1-ac50-f74d104dd36c None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:17.355120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a756e71-8b34-42d1-ac50-f74d104dd36c None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:17.355741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a756e71-8b34-42d1-ac50-f74d104dd36c None admin] RBAC: Adding query filter params (name=placement) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:17.356039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a756e71-8b34-42d1-ac50-f74d104dd36c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:17.357415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a756e71-8b34-42d1-ac50-f74d104dd36c None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:17.362310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 643/1291] 10.222.0.61 () {56 vars in 1116 bytes} [Thu Sep 29 02:03:17 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:17.368429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:17.481649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3089bfdd-bc60-4619-9271-7cfc1d3f5d1e None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:17.517615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ebb6e02-5c8e-49f4-8f3b-700b1edf41e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:17.518230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb6e02-5c8e-49f4-8f3b-700b1edf41e3 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:17.518508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb6e02-5c8e-49f4-8f3b-700b1edf41e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:17.518775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ebb6e02-5c8e-49f4-8f3b-700b1edf41e3 None admin] PATH_INFO: `/v3/services` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:17.519401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebb6e02-5c8e-49f4-8f3b-700b1edf41e3 None admin] RBAC: Adding query filter params (type=placement) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:17.519708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebb6e02-5c8e-49f4-8f3b-700b1edf41e3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:17.521069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ebb6e02-5c8e-49f4-8f3b-700b1edf41e3 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:17.525605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 649/1292] 10.222.0.61 () {56 vars in 1116 bytes} [Thu Sep 29 02:03:17 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:19.650678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d01d9c-4b8b-4c9c-a068-193219f77112 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:19.651219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d01d9c-4b8b-4c9c-a068-193219f77112 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:19.651219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d01d9c-4b8b-4c9c-a068-193219f77112 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:19.653283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 644/1293] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:03:19 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:19.661345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa98194-88a8-4ed9-9468-7c69cbd69500 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:19.661794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa98194-88a8-4ed9-9468-7c69cbd69500 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:19.661883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa98194-88a8-4ed9-9468-7c69cbd69500 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:19.682711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5fa98194-88a8-4ed9-9468-7c69cbd69500 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:19.683906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5fa98194-88a8-4ed9-9468-7c69cbd69500 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:19.714170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:19.753852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 650/1294] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:19 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 29 02:03:20.025680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d01d9c-4b8b-4c9c-a068-193219f77112 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:20.066073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:20.142901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d01d9c-4b8b-4c9c-a068-193219f77112 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:20.177623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e9daca8-ada1-4c3b-86c2-8dbe0c047a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:20.178843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9daca8-ada1-4c3b-86c2-8dbe0c047a49 None admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:20.179127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9daca8-ada1-4c3b-86c2-8dbe0c047a49 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:20.179386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e9daca8-ada1-4c3b-86c2-8dbe0c047a49 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:20.180149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9daca8-ada1-4c3b-86c2-8dbe0c047a49 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:20.181908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e9daca8-ada1-4c3b-86c2-8dbe0c047a49 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:20.200294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 645/1295] 10.222.0.61 () {60 vars in 1141 bytes} [Thu Sep 29 02:03:20 2022] POST /identity/v3/services => generated 246 bytes in 175 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:03:22.344593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194bcc81-baf4-4222-becc-98e9b9a537da None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:22.345527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194bcc81-baf4-4222-becc-98e9b9a537da None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:22.345527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194bcc81-baf4-4222-becc-98e9b9a537da None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:22.347649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 651/1296] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:03:22 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:22.354058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656bb9c0-d95f-43f7-bb8a-0430778c80b0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:22.354815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656bb9c0-d95f-43f7-bb8a-0430778c80b0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:22.354971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656bb9c0-d95f-43f7-bb8a-0430778c80b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:22.380054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-656bb9c0-d95f-43f7-bb8a-0430778c80b0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:22.381222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-656bb9c0-d95f-43f7-bb8a-0430778c80b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:22.458621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 646/1297] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:22 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:03:22.740398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194bcc81-baf4-4222-becc-98e9b9a537da None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:22.858424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194bcc81-baf4-4222-becc-98e9b9a537da None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:22.893730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:22.894374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:22.894638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:22.894903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:22.895473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:22.896832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:22.905790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:22.907048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:22.908390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 652/1298] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:03:22 2022] GET /identity/v3/services/placement => generated 90 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:22.914642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656bb9c0-d95f-43f7-bb8a-0430778c80b0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:22.934492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:23.016472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-656bb9c0-d95f-43f7-bb8a-0430778c80b0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:23.052803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:23.053738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:23.054041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:23.054301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:23.054919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:23.055214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:23.056824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:23.061923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 647/1299] 10.222.0.61 () {56 vars in 1116 bytes} [Thu Sep 29 02:03:22 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:23.068126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:23.113148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:23.197408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32d2771-4405-462d-9be6-580c4bee3349 None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:23.235035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60b26857-f96c-4c65-9a15-df8c9083eda8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:23.235641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b26857-f96c-4c65-9a15-df8c9083eda8 None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:23.235916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b26857-f96c-4c65-9a15-df8c9083eda8 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:23.236186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60b26857-f96c-4c65-9a15-df8c9083eda8 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:23.236884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b26857-f96c-4c65-9a15-df8c9083eda8 None admin] RBAC: Adding query filter params (interface=public, service_id=f906adf33a2a4314802ce9137596faab, region_id=RegionOne) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:23.237186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b26857-f96c-4c65-9a15-df8c9083eda8 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:23.238725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60b26857-f96c-4c65-9a15-df8c9083eda8 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:23.245796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 653/1300] 10.222.0.61 () {56 vars in 1251 bytes} [Thu Sep 29 02:03:23 2022] GET /identity/v3/endpoints?service_id=f906adf33a2a4314802ce9137596faab&interface=public®ion_id=RegionOne => generated 194 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:03:23.252668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:23.342363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:23.364333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84863db2-b72e-4fa9-8dc7-9fecca6dfae8 None admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:23.399975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447c995e-cba8-4a4a-8073-e04f2b7518d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:23.400599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c995e-cba8-4a4a-8073-e04f2b7518d4 None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:23.400871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c995e-cba8-4a4a-8073-e04f2b7518d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:23.401137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447c995e-cba8-4a4a-8073-e04f2b7518d4 None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:23.401735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447c995e-cba8-4a4a-8073-e04f2b7518d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:23.403147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447c995e-cba8-4a4a-8073-e04f2b7518d4 None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:23.410034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 648/1301] 10.222.0.61 () {56 vars in 1087 bytes} [Thu Sep 29 02:03:23 2022] GET /identity/v3/services => generated 2191 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:03:25.541843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23bb96-4fc4-4126-a0a6-de17de223dce None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:25.542721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23bb96-4fc4-4126-a0a6-de17de223dce None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:25.542721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db23bb96-4fc4-4126-a0a6-de17de223dce None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:25.544751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 654/1302] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:03:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:03:25.553284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5935952-830f-4838-841b-4f39ef85f77b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:25.554058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5935952-830f-4838-841b-4f39ef85f77b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:25.554215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5935952-830f-4838-841b-4f39ef85f77b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:25.574223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a5935952-830f-4838-841b-4f39ef85f77b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:03:25.575420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a5935952-830f-4838-841b-4f39ef85f77b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:03:25.648149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 649/1303] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:03:25 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 29 02:03:25.914564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23bb96-4fc4-4126-a0a6-de17de223dce None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:26.028482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:26.036669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db23bb96-4fc4-4126-a0a6-de17de223dce None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:26.071741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:26.072373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:26.072639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:26.072887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] PATH_INFO: `/v3/services/placement` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:26.073455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:26.074839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 29 02:03:26.083607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 29 02:03:26.084762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.get_or_create( Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application with Lock( Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self._enter() Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application return self.creator() Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application created_value = creator( Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: ERROR keystone.server.flask.application Sep 29 02:03:26.085872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 655/1304] 10.222.0.61 () {56 vars in 1117 bytes} [Thu Sep 29 02:03:25 2022] GET /identity/v3/services/placement => generated 90 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 29 02:03:26.091296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5935952-830f-4838-841b-4f39ef85f77b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:26.180095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:26.204528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5935952-830f-4838-841b-4f39ef85f77b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:26.239907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e68c5c2-bfe9-4274-85de-ad65dba36ddc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:26.240507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68c5c2-bfe9-4274-85de-ad65dba36ddc None admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:26.240779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68c5c2-bfe9-4274-85de-ad65dba36ddc None admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:26.241049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e68c5c2-bfe9-4274-85de-ad65dba36ddc None admin] PATH_INFO: `/v3/services` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:26.241671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e68c5c2-bfe9-4274-85de-ad65dba36ddc None admin] RBAC: Adding query filter params (name=placement) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:03:26.241976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e68c5c2-bfe9-4274-85de-ad65dba36ddc None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:26.243557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e68c5c2-bfe9-4274-85de-ad65dba36ddc None admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:26.248863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 650/1305] 10.222.0.61 () {56 vars in 1116 bytes} [Thu Sep 29 02:03:26 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:03:26.255076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:03:26.371521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdaf92ac-dbc5-4c24-a69b-89083ebc76fb None admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:03:26.408835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0a901a1-bb86-4f17-ad88-d8c85fa9fbc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:03:26.409686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a901a1-bb86-4f17-ad88-d8c85fa9fbc5 None admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:03:26.410079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a901a1-bb86-4f17-ad88-d8c85fa9fbc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:03:26.410470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0a901a1-bb86-4f17-ad88-d8c85fa9fbc5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:03:26.411453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a901a1-bb86-4f17-ad88-d8c85fa9fbc5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:03:26.413487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0a901a1-bb86-4f17-ad88-d8c85fa9fbc5 None admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:03:26.443988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 656/1306] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:03:26 2022] POST /identity/v3/endpoints => generated 331 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:03:36.245694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=30 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:03:36.439091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:31.139567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1318c5d-ad6d-4415-b1c5-2e26f34c7309 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:31.140038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1318c5d-ad6d-4415-b1c5-2e26f34c7309 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:31.140129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1318c5d-ad6d-4415-b1c5-2e26f34c7309 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:31.146917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8420913-f99e-4ec0-8349-04de969b2b27 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:31.147314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8420913-f99e-4ec0-8349-04de969b2b27 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:31.147394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8420913-f99e-4ec0-8349-04de969b2b27 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:31.150339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 651/1307] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:31 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:31.262422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b8420913-f99e-4ec0-8349-04de969b2b27 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:31.263615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b8420913-f99e-4ec0-8349-04de969b2b27 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:31.376217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 657/1308] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 230 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:04:31.667430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b89ce44-576e-48da-bb38-8ab7cf47fe3b None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:31.667766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b89ce44-576e-48da-bb38-8ab7cf47fe3b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:31.667988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b89ce44-576e-48da-bb38-8ab7cf47fe3b None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:31.669541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 652/1309] 10.222.0.61 () {54 vars in 964 bytes} [Thu Sep 29 02:04:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:31.675589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0ae386-5f5a-4bfd-be37-0a2ef5ae0125 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:31.675931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0ae386-5f5a-4bfd-be37-0a2ef5ae0125 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:31.676267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a0ae386-5f5a-4bfd-be37-0a2ef5ae0125 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:31.762920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9a0ae386-5f5a-4bfd-be37-0a2ef5ae0125 None None] MFA Rules not processed for user `cc2f041fe7194a3da199517b4cdbd265`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:31.764114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9a0ae386-5f5a-4bfd-be37-0a2ef5ae0125 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:31.786336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:31.845361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 658/1310] 10.222.0.61 () {58 vars in 1061 bytes} [Thu Sep 29 02:04:31 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:31.854726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b89ce44-576e-48da-bb38-8ab7cf47fe3b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:31.856955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2b89ce44-576e-48da-bb38-8ab7cf47fe3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:31.967362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b89ce44-576e-48da-bb38-8ab7cf47fe3b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:32.046172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c679988b-1b42-465c-8957-271df0e4aa32 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:32.046780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679988b-1b42-465c-8957-271df0e4aa32 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:32.047030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679988b-1b42-465c-8957-271df0e4aa32 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:32.047310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c679988b-1b42-465c-8957-271df0e4aa32 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:32.048884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c679988b-1b42-465c-8957-271df0e4aa32 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:32.108893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:32.173993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c679988b-1b42-465c-8957-271df0e4aa32 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:32.175897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c679988b-1b42-465c-8957-271df0e4aa32 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:32.199988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 653/1311] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:04:34.618048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ead4f4-db52-46d2-a35e-0f15ea9c117e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:34.618627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ead4f4-db52-46d2-a35e-0f15ea9c117e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:34.618627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ead4f4-db52-46d2-a35e-0f15ea9c117e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:34.619916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 659/1312] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:34.626428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df4682a-04f0-486c-a4b2-d524f30fd29c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:34.626881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df4682a-04f0-486c-a4b2-d524f30fd29c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:34.626975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df4682a-04f0-486c-a4b2-d524f30fd29c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:34.712937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3df4682a-04f0-486c-a4b2-d524f30fd29c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:34.714165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3df4682a-04f0-486c-a4b2-d524f30fd29c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:34.821320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 654/1313] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:35.128804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6be413-798a-4710-94f3-dee2753159a0 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:35.129034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6be413-798a-4710-94f3-dee2753159a0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:35.129340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6be413-798a-4710-94f3-dee2753159a0 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:35.130731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 660/1314] 10.222.0.61 () {54 vars in 964 bytes} [Thu Sep 29 02:04:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:35.136637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de10c3-b545-46aa-beb3-40a231846992 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:35.136941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de10c3-b545-46aa-beb3-40a231846992 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:35.137238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41de10c3-b545-46aa-beb3-40a231846992 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:35.149145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:35.218612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-41de10c3-b545-46aa-beb3-40a231846992 None None] MFA Rules not processed for user `cc2f041fe7194a3da199517b4cdbd265`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:35.219732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-41de10c3-b545-46aa-beb3-40a231846992 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:35.301152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 655/1315] 10.222.0.61 () {58 vars in 1061 bytes} [Thu Sep 29 02:04:35 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:35.310718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6be413-798a-4710-94f3-dee2753159a0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:35.312259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7a6be413-798a-4710-94f3-dee2753159a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:35.420043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6be413-798a-4710-94f3-dee2753159a0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:35.494298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:35.496782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053f1897-160b-44e2-bac8-3fc4b43dc51c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:35.497396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053f1897-160b-44e2-bac8-3fc4b43dc51c service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:35.497688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053f1897-160b-44e2-bac8-3fc4b43dc51c service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:35.497958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053f1897-160b-44e2-bac8-3fc4b43dc51c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:35.500068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-053f1897-160b-44e2-bac8-3fc4b43dc51c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:35.615557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053f1897-160b-44e2-bac8-3fc4b43dc51c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:35.617154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053f1897-160b-44e2-bac8-3fc4b43dc51c service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:35.635550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 661/1316] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:38.358025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14a55a-7511-45d5-a395-4bd7a7ebdfa0 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:38.358625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14a55a-7511-45d5-a395-4bd7a7ebdfa0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:38.358625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df14a55a-7511-45d5-a395-4bd7a7ebdfa0 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:38.359885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 656/1317] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:38.365560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdabaa17-dbde-4348-8140-093a8dbe73b7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:38.365963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdabaa17-dbde-4348-8140-093a8dbe73b7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:38.366203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdabaa17-dbde-4348-8140-093a8dbe73b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:38.439147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cdabaa17-dbde-4348-8140-093a8dbe73b7 None None] MFA Rules not processed for user `9564038bafec44429094752520f2166c`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:38.440348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cdabaa17-dbde-4348-8140-093a8dbe73b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:38.496703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:38.547797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 662/1318] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:38 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:38.832250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df14a55a-7511-45d5-a395-4bd7a7ebdfa0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:38.834822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-df14a55a-7511-45d5-a395-4bd7a7ebdfa0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:38.942719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df14a55a-7511-45d5-a395-4bd7a7ebdfa0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:38.945525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:39.017111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92fc9001-6432-46e8-881b-8e99c5576401 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:39.017712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc9001-6432-46e8-881b-8e99c5576401 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:39.017984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc9001-6432-46e8-881b-8e99c5576401 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:39.018283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92fc9001-6432-46e8-881b-8e99c5576401 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:39.019876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-92fc9001-6432-46e8-881b-8e99c5576401 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:39.141692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fc9001-6432-46e8-881b-8e99c5576401 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:39.143087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92fc9001-6432-46e8-881b-8e99c5576401 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:39.162702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 657/1319] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:38 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 332 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:41.872879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04977f73-178d-4cc0-a1c0-86cc109088d8 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:41.873431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04977f73-178d-4cc0-a1c0-86cc109088d8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:41.873431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04977f73-178d-4cc0-a1c0-86cc109088d8 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:41.875558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 663/1320] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:41 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:41.884256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab1bf7-166c-4ed2-845b-d2a8f7817ff1 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:41.884993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab1bf7-166c-4ed2-845b-d2a8f7817ff1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:41.885156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dab1bf7-166c-4ed2-845b-d2a8f7817ff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:41.934123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:41.961117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7dab1bf7-166c-4ed2-845b-d2a8f7817ff1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:41.962346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7dab1bf7-166c-4ed2-845b-d2a8f7817ff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:42.066367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 658/1321] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:41 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:42.350456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04977f73-178d-4cc0-a1c0-86cc109088d8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:42.352842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-04977f73-178d-4cc0-a1c0-86cc109088d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:42.468494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04977f73-178d-4cc0-a1c0-86cc109088d8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:42.549561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0092429-4b97-4c11-aaab-cefb3f8727c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:42.550127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0092429-4b97-4c11-aaab-cefb3f8727c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:42.550846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0092429-4b97-4c11-aaab-cefb3f8727c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:42.550846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0092429-4b97-4c11-aaab-cefb3f8727c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:42.553093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c0092429-4b97-4c11-aaab-cefb3f8727c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:42.562703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:42.675882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0092429-4b97-4c11-aaab-cefb3f8727c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:42.677332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0092429-4b97-4c11-aaab-cefb3f8727c6 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:42.702107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 664/1322] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:45.486974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a7a7ad-74c8-483b-9d43-fbb7fd4284a8 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:45.487591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a7a7ad-74c8-483b-9d43-fbb7fd4284a8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:45.487591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39a7a7ad-74c8-483b-9d43-fbb7fd4284a8 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:45.488865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 659/1323] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:45.494293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d5789-7485-4eca-a839-3d197ef3959d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:45.494737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d5789-7485-4eca-a839-3d197ef3959d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:45.494857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7d5789-7485-4eca-a839-3d197ef3959d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:45.570523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7f7d5789-7485-4eca-a839-3d197ef3959d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:45.571922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7f7d5789-7485-4eca-a839-3d197ef3959d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:45.636005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:45.679818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 665/1324] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:45.955982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a7a7ad-74c8-483b-9d43-fbb7fd4284a8 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:45.957515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-39a7a7ad-74c8-483b-9d43-fbb7fd4284a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:46.064352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39a7a7ad-74c8-483b-9d43-fbb7fd4284a8 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:46.077771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:46.140673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e77d17-afb6-4f30-a34a-ef0e638cbe97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:46.141283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e77d17-afb6-4f30-a34a-ef0e638cbe97 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:46.141560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e77d17-afb6-4f30-a34a-ef0e638cbe97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:46.141831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e77d17-afb6-4f30-a34a-ef0e638cbe97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:46.143500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-68e77d17-afb6-4f30-a34a-ef0e638cbe97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:46.260044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e77d17-afb6-4f30-a34a-ef0e638cbe97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:46.261500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e77d17-afb6-4f30-a34a-ef0e638cbe97 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:46.279782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 660/1325] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:48.493203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e80ff18-33b2-4fed-ad76-6d0a52dfd277 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:48.493753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e80ff18-33b2-4fed-ad76-6d0a52dfd277 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:48.493753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e80ff18-33b2-4fed-ad76-6d0a52dfd277 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:48.495009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 666/1326] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:48.500793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af565fb-c1bb-491c-a738-aeb0ea3dbd43 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:48.501221 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af565fb-c1bb-491c-a738-aeb0ea3dbd43 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:48.501310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af565fb-c1bb-491c-a738-aeb0ea3dbd43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:48.550995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:48.570753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9af565fb-c1bb-491c-a738-aeb0ea3dbd43 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:48.571966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9af565fb-c1bb-491c-a738-aeb0ea3dbd43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:48.675345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 661/1327] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:48.956319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e80ff18-33b2-4fed-ad76-6d0a52dfd277 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:48.957866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1e80ff18-33b2-4fed-ad76-6d0a52dfd277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:49.067689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e80ff18-33b2-4fed-ad76-6d0a52dfd277 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:49.145389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c98185d0-51c8-4039-bf7a-d354ebc672e3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:49.146024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98185d0-51c8-4039-bf7a-d354ebc672e3 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:49.146306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98185d0-51c8-4039-bf7a-d354ebc672e3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:49.146578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c98185d0-51c8-4039-bf7a-d354ebc672e3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:49.148194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c98185d0-51c8-4039-bf7a-d354ebc672e3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:49.166829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:49.261965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98185d0-51c8-4039-bf7a-d354ebc672e3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:49.263449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c98185d0-51c8-4039-bf7a-d354ebc672e3 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:49.282411 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 667/1328] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:51.532727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2bb6e-2fb6-49c7-8ebe-20e0e0d56cfd None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:51.533279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2bb6e-2fb6-49c7-8ebe-20e0e0d56cfd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:51.533279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2bb6e-2fb6-49c7-8ebe-20e0e0d56cfd None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:51.534582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 662/1329] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:51.540906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ac745-960a-442d-b365-a89019d0b2ce None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:51.541346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ac745-960a-442d-b365-a89019d0b2ce None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:51.541548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d7ac745-960a-442d-b365-a89019d0b2ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:51.613676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3d7ac745-960a-442d-b365-a89019d0b2ce None None] MFA Rules not processed for user `9564038bafec44429094752520f2166c`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:51.614916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3d7ac745-960a-442d-b365-a89019d0b2ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:51.684491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:51.727571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 668/1330] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:51 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:52.011219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef2bb6e-2fb6-49c7-8ebe-20e0e0d56cfd None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:52.012778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3ef2bb6e-2fb6-49c7-8ebe-20e0e0d56cfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:52.118928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef2bb6e-2fb6-49c7-8ebe-20e0e0d56cfd None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:52.140230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:52.194575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d37560-3476-469d-82a1-511bd0615584 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:52.195182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d37560-3476-469d-82a1-511bd0615584 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:52.195454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d37560-3476-469d-82a1-511bd0615584 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:52.195716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d37560-3476-469d-82a1-511bd0615584 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:52.197338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f8d37560-3476-469d-82a1-511bd0615584 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:52.321026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d37560-3476-469d-82a1-511bd0615584 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:52.322511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8d37560-3476-469d-82a1-511bd0615584 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:52.344018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 663/1331] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:52 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:54.915133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959e8e85-4040-4d09-a2bf-3704af5cde43 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:54.915726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959e8e85-4040-4d09-a2bf-3704af5cde43 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:54.915726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959e8e85-4040-4d09-a2bf-3704af5cde43 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:54.917719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 669/1332] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:54 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:54.924205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd43dcef-af2d-4417-bdfa-3fe6144a8c4e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:54.924650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd43dcef-af2d-4417-bdfa-3fe6144a8c4e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:54.924761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd43dcef-af2d-4417-bdfa-3fe6144a8c4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:55.000602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:55.003797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cd43dcef-af2d-4417-bdfa-3fe6144a8c4e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:55.005934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cd43dcef-af2d-4417-bdfa-3fe6144a8c4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:55.107622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 664/1333] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:55.403218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959e8e85-4040-4d09-a2bf-3704af5cde43 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:55.405162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-959e8e85-4040-4d09-a2bf-3704af5cde43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:55.528488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959e8e85-4040-4d09-a2bf-3704af5cde43 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:55.611120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0593889-aa80-4d6f-a6dd-095745291eb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:55.611730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0593889-aa80-4d6f-a6dd-095745291eb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:55.612000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0593889-aa80-4d6f-a6dd-095745291eb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:55.612271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0593889-aa80-4d6f-a6dd-095745291eb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:55.613859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e0593889-aa80-4d6f-a6dd-095745291eb7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:55.643499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:55.730687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0593889-aa80-4d6f-a6dd-095745291eb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:55.732122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0593889-aa80-4d6f-a6dd-095745291eb7 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:55.753539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 670/1334] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:04:57.925837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12905043-e337-4c3a-a1a7-15bac162ec9f None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:57.926728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12905043-e337-4c3a-a1a7-15bac162ec9f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:57.926728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12905043-e337-4c3a-a1a7-15bac162ec9f None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:57.928793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 665/1335] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:04:57 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:04:57.934428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd4e9e2-2fac-452e-a829-daa13f873c00 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:57.934916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd4e9e2-2fac-452e-a829-daa13f873c00 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:57.934916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd4e9e2-2fac-452e-a829-daa13f873c00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:58.001675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5cd4e9e2-2fac-452e-a829-daa13f873c00 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:04:58.002893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5cd4e9e2-2fac-452e-a829-daa13f873c00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:58.085150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:58.112949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 671/1336] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:04:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:04:58.398829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12905043-e337-4c3a-a1a7-15bac162ec9f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:04:58.401341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-12905043-e337-4c3a-a1a7-15bac162ec9f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:58.513336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12905043-e337-4c3a-a1a7-15bac162ec9f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:04:58.550703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:04:58.599333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc083aa3-03bd-4447-b7fd-98cb8867403e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:04:58.599951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc083aa3-03bd-4447-b7fd-98cb8867403e service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:04:58.600226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc083aa3-03bd-4447-b7fd-98cb8867403e service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:04:58.600497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc083aa3-03bd-4447-b7fd-98cb8867403e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:04:58.602127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bc083aa3-03bd-4447-b7fd-98cb8867403e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:04:58.721867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc083aa3-03bd-4447-b7fd-98cb8867403e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:04:58.723370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc083aa3-03bd-4447-b7fd-98cb8867403e service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:04:58.742018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 666/1337] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:04:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:01.411121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b517a29-1c20-4d5c-95b5-7c983927470c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:01.412033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b517a29-1c20-4d5c-95b5-7c983927470c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:01.412033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b517a29-1c20-4d5c-95b5-7c983927470c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:01.414090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 672/1338] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:01.423060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978255b-ce92-4ed5-ac40-b29df772804e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:01.423504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978255b-ce92-4ed5-ac40-b29df772804e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:01.423636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2978255b-ce92-4ed5-ac40-b29df772804e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:01.500519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2978255b-ce92-4ed5-ac40-b29df772804e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:01.501631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2978255b-ce92-4ed5-ac40-b29df772804e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:01.510588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:01.603038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 667/1339] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:01 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:01.886253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b517a29-1c20-4d5c-95b5-7c983927470c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:01.888651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9b517a29-1c20-4d5c-95b5-7c983927470c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:02.000671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b517a29-1c20-4d5c-95b5-7c983927470c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:02.098872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9659f6-7f5b-4f51-8f8b-e21688c582bc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:02.099479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9659f6-7f5b-4f51-8f8b-e21688c582bc service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:02.099751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9659f6-7f5b-4f51-8f8b-e21688c582bc service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:02.100017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9659f6-7f5b-4f51-8f8b-e21688c582bc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:02.101648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-dc9659f6-7f5b-4f51-8f8b-e21688c582bc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:02.142217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:02.222047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9659f6-7f5b-4f51-8f8b-e21688c582bc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:02.223565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9659f6-7f5b-4f51-8f8b-e21688c582bc service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:02.245933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 673/1340] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:01 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 361 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:06.449331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c078f29-235c-4976-8016-7027bb39b030 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:06.449832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c078f29-235c-4976-8016-7027bb39b030 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:06.449943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c078f29-235c-4976-8016-7027bb39b030 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:06.451404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 668/1341] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:06.457183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd99632c-bb79-480a-b41c-fafec43e5e42 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:06.457611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd99632c-bb79-480a-b41c-fafec43e5e42 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:06.457701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd99632c-bb79-480a-b41c-fafec43e5e42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:06.530307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fd99632c-bb79-480a-b41c-fafec43e5e42 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:06.531929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fd99632c-bb79-480a-b41c-fafec43e5e42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:06.629843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:06.642220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 674/1342] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:05:06.946150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c078f29-235c-4976-8016-7027bb39b030 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:06.947681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8c078f29-235c-4976-8016-7027bb39b030 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:07.071491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c078f29-235c-4976-8016-7027bb39b030 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:07.122200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:07.161431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8744c18e-6138-46bd-b092-d387efbe8ac2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:07.162041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8744c18e-6138-46bd-b092-d387efbe8ac2 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:07.162331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8744c18e-6138-46bd-b092-d387efbe8ac2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:07.162597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8744c18e-6138-46bd-b092-d387efbe8ac2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:07.164211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8744c18e-6138-46bd-b092-d387efbe8ac2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:07.304497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8744c18e-6138-46bd-b092-d387efbe8ac2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:07.305866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8744c18e-6138-46bd-b092-d387efbe8ac2 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:07.332516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 669/1343] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:10.147079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4fed02-de6f-4cf4-9cc2-dc443cc97545 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:10.147682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4fed02-de6f-4cf4-9cc2-dc443cc97545 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:10.147682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a4fed02-de6f-4cf4-9cc2-dc443cc97545 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:10.148979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 675/1344] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:10.154281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcf5c11-5ef2-48e3-894b-6804148b0785 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:10.154731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcf5c11-5ef2-48e3-894b-6804148b0785 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:10.154827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcf5c11-5ef2-48e3-894b-6804148b0785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:10.229343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1bcf5c11-5ef2-48e3-894b-6804148b0785 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:10.230769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1bcf5c11-5ef2-48e3-894b-6804148b0785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:10.259945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:10.345749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 670/1345] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:10 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:10.627105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4fed02-de6f-4cf4-9cc2-dc443cc97545 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:10.628741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6a4fed02-de6f-4cf4-9cc2-dc443cc97545 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:10.734283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a4fed02-de6f-4cf4-9cc2-dc443cc97545 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:10.809213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bdcdd07-8994-408a-8fc0-1004ad1e242f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:10.809818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcdd07-8994-408a-8fc0-1004ad1e242f service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:10.810135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcdd07-8994-408a-8fc0-1004ad1e242f service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:10.810357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bdcdd07-8994-408a-8fc0-1004ad1e242f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:10.812019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9bdcdd07-8994-408a-8fc0-1004ad1e242f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:10.856645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:10.928583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdcdd07-8994-408a-8fc0-1004ad1e242f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:10.930054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bdcdd07-8994-408a-8fc0-1004ad1e242f service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:10.949827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 676/1346] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:10 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:15.419612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cff8bcd-3bfc-4cae-a529-63850462e599 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:15.420145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cff8bcd-3bfc-4cae-a529-63850462e599 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:15.420145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cff8bcd-3bfc-4cae-a529-63850462e599 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:15.421782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 671/1347] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:15.427341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63029808-5732-45ed-8479-0f9955436f34 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:15.427913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63029808-5732-45ed-8479-0f9955436f34 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:15.427913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63029808-5732-45ed-8479-0f9955436f34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:15.499884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-63029808-5732-45ed-8479-0f9955436f34 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:15.501128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-63029808-5732-45ed-8479-0f9955436f34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:15.598879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:15.602541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 677/1348] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:15.888588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cff8bcd-3bfc-4cae-a529-63850462e599 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:15.890978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8cff8bcd-3bfc-4cae-a529-63850462e599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:16.002613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cff8bcd-3bfc-4cae-a529-63850462e599 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:16.060905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:16.094283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43677cd4-16b7-4c0d-9ca7-7c6b4635413a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:16.094883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43677cd4-16b7-4c0d-9ca7-7c6b4635413a service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:16.095152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43677cd4-16b7-4c0d-9ca7-7c6b4635413a service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:16.095418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43677cd4-16b7-4c0d-9ca7-7c6b4635413a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:16.097001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-43677cd4-16b7-4c0d-9ca7-7c6b4635413a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:16.214893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43677cd4-16b7-4c0d-9ca7-7c6b4635413a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:16.216348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43677cd4-16b7-4c0d-9ca7-7c6b4635413a service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:16.235466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 672/1349] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:21.225317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e53912-9d5c-407b-9cf2-437b3a3f6efc None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:21.225317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e53912-9d5c-407b-9cf2-437b3a3f6efc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:21.226348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e53912-9d5c-407b-9cf2-437b3a3f6efc None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:21.228224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 678/1350] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:05:21 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:21.233337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0924c618-2fde-494b-beb6-180013abce90 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:21.233771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0924c618-2fde-494b-beb6-180013abce90 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:21.233868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0924c618-2fde-494b-beb6-180013abce90 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:21.338067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0924c618-2fde-494b-beb6-180013abce90 None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:21.339371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0924c618-2fde-494b-beb6-180013abce90 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:21.403284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:21.528324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 673/1351] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:05:21 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:25.829721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee047da5-4158-48f9-8c71-c4b80dcc7e17 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:25.830367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee047da5-4158-48f9-8c71-c4b80dcc7e17 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:25.830367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee047da5-4158-48f9-8c71-c4b80dcc7e17 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:25.831859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 679/1352] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:05:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:25.837026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66229e1e-cc34-4d69-8ab7-5b8a49483b4c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:25.837487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66229e1e-cc34-4d69-8ab7-5b8a49483b4c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:25.837760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66229e1e-cc34-4d69-8ab7-5b8a49483b4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:25.927903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-66229e1e-cc34-4d69-8ab7-5b8a49483b4c None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:25.929093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-66229e1e-cc34-4d69-8ab7-5b8a49483b4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:26.006293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:26.043596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 674/1353] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:05:25 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:27.091344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e853adc0-4828-42cf-8653-6bc8fd894b76 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:27.091904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e853adc0-4828-42cf-8653-6bc8fd894b76 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:27.091904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e853adc0-4828-42cf-8653-6bc8fd894b76 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:27.093241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 680/1354] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:05:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:27.098624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cdd933-6460-452b-916b-863fdc16e3ed None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:27.099069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cdd933-6460-452b-916b-863fdc16e3ed None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:27.099232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79cdd933-6460-452b-916b-863fdc16e3ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:27.179549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-79cdd933-6460-452b-916b-863fdc16e3ed None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:27.180808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-79cdd933-6460-452b-916b-863fdc16e3ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:27.291181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 675/1355] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:05:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:30.918277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0cb8c-f81d-4809-b6e8-0aab35310664 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:30.918853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0cb8c-f81d-4809-b6e8-0aab35310664 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:30.918962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fd0cb8c-f81d-4809-b6e8-0aab35310664 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:30.920396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 681/1356] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:30.926111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bc1fc9-c568-46de-9db2-fd98e5e4767e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:30.926603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bc1fc9-c568-46de-9db2-fd98e5e4767e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:30.926708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bc1fc9-c568-46de-9db2-fd98e5e4767e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:30.966475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:31.000068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-90bc1fc9-c568-46de-9db2-fd98e5e4767e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:31.001325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-90bc1fc9-c568-46de-9db2-fd98e5e4767e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:31.103087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 676/1357] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:31.401716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb28bf2-4da2-4cdf-9cab-50f296e9af0d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:31.402272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb28bf2-4da2-4cdf-9cab-50f296e9af0d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:31.402509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb28bf2-4da2-4cdf-9cab-50f296e9af0d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:31.403839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 682/1358] 10.222.0.61 () {54 vars in 961 bytes} [Thu Sep 29 02:05:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:31.413003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6507a2-7a8a-457f-ba1c-c9b2397ced6c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:31.413292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6507a2-7a8a-457f-ba1c-c9b2397ced6c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:31.413571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6507a2-7a8a-457f-ba1c-c9b2397ced6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:31.501101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2c6507a2-7a8a-457f-ba1c-c9b2397ced6c None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:31.502254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2c6507a2-7a8a-457f-ba1c-c9b2397ced6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:31.560665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:31.654220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 677/1359] 10.222.0.61 () {58 vars in 1058 bytes} [Thu Sep 29 02:05:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:31.662753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb28bf2-4da2-4cdf-9cab-50f296e9af0d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:31.664635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fbb28bf2-4da2-4cdf-9cab-50f296e9af0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:31.834127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb28bf2-4da2-4cdf-9cab-50f296e9af0d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:31.910567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:31.956632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31bf4f1c-9a8e-4ee0-ad7f-2aecc718ece2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:31.958344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf4f1c-9a8e-4ee0-ad7f-2aecc718ece2 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:31.958719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf4f1c-9a8e-4ee0-ad7f-2aecc718ece2 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:31.959114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31bf4f1c-9a8e-4ee0-ad7f-2aecc718ece2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:31.960843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-31bf4f1c-9a8e-4ee0-ad7f-2aecc718ece2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:32.064714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bf4f1c-9a8e-4ee0-ad7f-2aecc718ece2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:32.066551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31bf4f1c-9a8e-4ee0-ad7f-2aecc718ece2 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:32.086402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 683/1360] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:05:34.461068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c2caa6-bf7e-4587-8d48-2750bc578171 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:34.461650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c2caa6-bf7e-4587-8d48-2750bc578171 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:34.461650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c2caa6-bf7e-4587-8d48-2750bc578171 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:34.463157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 678/1361] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:34.470037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22735f90-e8a2-41f5-8f83-beeaa2368740 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:34.470037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22735f90-e8a2-41f5-8f83-beeaa2368740 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:34.470037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22735f90-e8a2-41f5-8f83-beeaa2368740 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:34.550992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-22735f90-e8a2-41f5-8f83-beeaa2368740 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:34.552372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-22735f90-e8a2-41f5-8f83-beeaa2368740 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:34.593503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:34.662117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 684/1362] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:34.948782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c2caa6-bf7e-4587-8d48-2750bc578171 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:34.950516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f0c2caa6-bf7e-4587-8d48-2750bc578171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:35.127222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c2caa6-bf7e-4587-8d48-2750bc578171 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:35.138407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:35.253651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eaf5121-57c9-4716-a95c-e30da23bb4e7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:35.254318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5121-57c9-4716-a95c-e30da23bb4e7 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:35.254580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5121-57c9-4716-a95c-e30da23bb4e7 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:35.254868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eaf5121-57c9-4716-a95c-e30da23bb4e7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:35.256471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2eaf5121-57c9-4716-a95c-e30da23bb4e7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:35.348482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf5121-57c9-4716-a95c-e30da23bb4e7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:35.350170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eaf5121-57c9-4716-a95c-e30da23bb4e7 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:35.369393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 679/1363] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 421 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:37.896814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bcf527-c188-4550-8c32-bf85a1be7bf6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:37.897393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bcf527-c188-4550-8c32-bf85a1be7bf6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:37.897393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62bcf527-c188-4550-8c32-bf85a1be7bf6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:37.898901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 685/1364] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:37.904114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f0e941-74f6-4291-96e3-1ca9ef805319 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:37.904540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f0e941-74f6-4291-96e3-1ca9ef805319 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:37.904664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f0e941-74f6-4291-96e3-1ca9ef805319 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:37.920907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:37.977112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-39f0e941-74f6-4291-96e3-1ca9ef805319 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:37.978341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-39f0e941-74f6-4291-96e3-1ca9ef805319 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:38.083425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 680/1365] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:38.373067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2e5a03-a379-4db7-8045-b32d6433a583 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:38.373300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2e5a03-a379-4db7-8045-b32d6433a583 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:38.373573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2e5a03-a379-4db7-8045-b32d6433a583 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:38.375131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 686/1366] 10.222.0.61 () {54 vars in 961 bytes} [Thu Sep 29 02:05:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:38.380483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb806117-ab5f-4909-8906-53a8de5a72cc None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:38.380859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb806117-ab5f-4909-8906-53a8de5a72cc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:38.381353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb806117-ab5f-4909-8906-53a8de5a72cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:38.467312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-bb806117-ab5f-4909-8906-53a8de5a72cc None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:38.468629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bb806117-ab5f-4909-8906-53a8de5a72cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:38.511495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:38.612189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 681/1367] 10.222.0.61 () {58 vars in 1058 bytes} [Thu Sep 29 02:05:38 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 233 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:38.620469 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2e5a03-a379-4db7-8045-b32d6433a583 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:38.622082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8b2e5a03-a379-4db7-8045-b32d6433a583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:38.788704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2e5a03-a379-4db7-8045-b32d6433a583 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:38.811743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:38.924495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-134dbec2-767d-41f6-8f43-03842d78c83f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:38.925214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134dbec2-767d-41f6-8f43-03842d78c83f service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:38.925363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134dbec2-767d-41f6-8f43-03842d78c83f service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:38.925638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-134dbec2-767d-41f6-8f43-03842d78c83f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:38.927363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-134dbec2-767d-41f6-8f43-03842d78c83f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:38.996090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d83d8aa-d865-457c-a44f-1172e4db00e1 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:38.996720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d83d8aa-d865-457c-a44f-1172e4db00e1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:38.996720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d83d8aa-d865-457c-a44f-1172e4db00e1 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:38.998169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 682/1368] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:05:39.004097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19b864-4628-4279-b5e3-7eab7ccbd48f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:39.004510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19b864-4628-4279-b5e3-7eab7ccbd48f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:39.004918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb19b864-4628-4279-b5e3-7eab7ccbd48f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:39.039126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134dbec2-767d-41f6-8f43-03842d78c83f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:39.040657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-134dbec2-767d-41f6-8f43-03842d78c83f service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:39.060605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 687/1369] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:39.077292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-eb19b864-4628-4279-b5e3-7eab7ccbd48f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:39.078673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-eb19b864-4628-4279-b5e3-7eab7ccbd48f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:39.196475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:39.214036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 683/1370] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:05:39.556863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaa62e1-59ca-4da6-875e-28cd2f9904a7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:39.557253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaa62e1-59ca-4da6-875e-28cd2f9904a7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:39.557609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aaa62e1-59ca-4da6-875e-28cd2f9904a7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:39.559292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 688/1371] 10.222.0.61 () {54 vars in 963 bytes} [Thu Sep 29 02:05:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:39.565144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5611efc9-8499-43eb-9d73-0dde3615c4a8 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:39.565517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5611efc9-8499-43eb-9d73-0dde3615c4a8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:39.565889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5611efc9-8499-43eb-9d73-0dde3615c4a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:39.647546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5611efc9-8499-43eb-9d73-0dde3615c4a8 None None] MFA Rules not processed for user `36c4d127aac743baaeb86157dbbeb689`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:39.648842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5611efc9-8499-43eb-9d73-0dde3615c4a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:39.739607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 684/1372] 10.222.0.61 () {58 vars in 1060 bytes} [Thu Sep 29 02:05:39 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:39.747510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaa62e1-59ca-4da6-875e-28cd2f9904a7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:39.749025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5aaa62e1-59ca-4da6-875e-28cd2f9904a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:39.780428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:39.866527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aaa62e1-59ca-4da6-875e-28cd2f9904a7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:39.957690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-905232ae-aaac-4626-8d22-13ba76d25403 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:39.958354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905232ae-aaac-4626-8d22-13ba76d25403 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:39.958647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905232ae-aaac-4626-8d22-13ba76d25403 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:39.958918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-905232ae-aaac-4626-8d22-13ba76d25403 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:39.960555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-905232ae-aaac-4626-8d22-13ba76d25403 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:40.062169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:40.126922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905232ae-aaac-4626-8d22-13ba76d25403 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:40.128586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-905232ae-aaac-4626-8d22-13ba76d25403 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:40.148690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 689/1373] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:41.307333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a415b7-c046-4d0d-8d2d-a09e6ea4f27a None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:41.307887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a415b7-c046-4d0d-8d2d-a09e6ea4f27a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:41.307887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18a415b7-c046-4d0d-8d2d-a09e6ea4f27a None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:41.309117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 685/1374] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:41.315295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4036c1-606c-4397-9199-a51d1548792a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:41.315748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4036c1-606c-4397-9199-a51d1548792a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:41.315861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4036c1-606c-4397-9199-a51d1548792a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:41.384490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-dc4036c1-606c-4397-9199-a51d1548792a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:41.385695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-dc4036c1-606c-4397-9199-a51d1548792a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:41.488356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 690/1375] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:41.791217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a415b7-c046-4d0d-8d2d-a09e6ea4f27a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:41.792734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-18a415b7-c046-4d0d-8d2d-a09e6ea4f27a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:41.883160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:41.945470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18a415b7-c046-4d0d-8d2d-a09e6ea4f27a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:42.062971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6840fce-aaac-4ac6-b99c-b86779600792 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:42.063634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6840fce-aaac-4ac6-b99c-b86779600792 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:42.063908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6840fce-aaac-4ac6-b99c-b86779600792 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:42.064174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6840fce-aaac-4ac6-b99c-b86779600792 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:42.065767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b6840fce-aaac-4ac6-b99c-b86779600792 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:42.119338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:42.161845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6840fce-aaac-4ac6-b99c-b86779600792 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:42.163498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6840fce-aaac-4ac6-b99c-b86779600792 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:42.186757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 686/1376] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 396 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:05:44.398659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6e1f2b-9c80-4521-a5ef-a8aaa88c770c None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:44.399268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6e1f2b-9c80-4521-a5ef-a8aaa88c770c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:44.399268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6e1f2b-9c80-4521-a5ef-a8aaa88c770c None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:44.400878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 691/1377] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:44.407080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9799cb5-7460-4612-82a4-19633b0c8ef2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:44.408172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9799cb5-7460-4612-82a4-19633b0c8ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:44.408730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9799cb5-7460-4612-82a4-19633b0c8ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:44.484087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a9799cb5-7460-4612-82a4-19633b0c8ef2 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:44.485589 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a9799cb5-7460-4612-82a4-19633b0c8ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:44.596851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 687/1378] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:44.884039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6e1f2b-9c80-4521-a5ef-a8aaa88c770c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:44.885535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2d6e1f2b-9c80-4521-a5ef-a8aaa88c770c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:44.925448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:45.039405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6e1f2b-9c80-4521-a5ef-a8aaa88c770c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:45.150989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:45.155913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b205e71-37b6-4749-b2cd-1e134f93c6e0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:45.156563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b205e71-37b6-4749-b2cd-1e134f93c6e0 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:45.156772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b205e71-37b6-4749-b2cd-1e134f93c6e0 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:45.157102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b205e71-37b6-4749-b2cd-1e134f93c6e0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:45.159463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2b205e71-37b6-4749-b2cd-1e134f93c6e0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:45.258779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b205e71-37b6-4749-b2cd-1e134f93c6e0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:45.260370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b205e71-37b6-4749-b2cd-1e134f93c6e0 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:45.279187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 692/1379] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 395 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:47.523817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd17472-5dce-4683-98ba-f27170798be8 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:47.524259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd17472-5dce-4683-98ba-f27170798be8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:47.524259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd17472-5dce-4683-98ba-f27170798be8 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:47.526261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 688/1380] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:47.535139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537901e9-dffe-48a9-847b-2ccff6d76462 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:47.535779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537901e9-dffe-48a9-847b-2ccff6d76462 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:47.535779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537901e9-dffe-48a9-847b-2ccff6d76462 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:47.627679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-537901e9-dffe-48a9-847b-2ccff6d76462 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:47.628921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-537901e9-dffe-48a9-847b-2ccff6d76462 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:47.705539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:47.739377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 693/1381] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:48.030722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd17472-5dce-4683-98ba-f27170798be8 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:48.032970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bfd17472-5dce-4683-98ba-f27170798be8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:48.066655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:48.194836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd17472-5dce-4683-98ba-f27170798be8 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:48.296163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:48.312751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff60a9f-a4e3-4286-b81b-9a29461ce642 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:48.313437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff60a9f-a4e3-4286-b81b-9a29461ce642 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:48.313793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff60a9f-a4e3-4286-b81b-9a29461ce642 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:48.314101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff60a9f-a4e3-4286-b81b-9a29461ce642 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:48.315806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-fff60a9f-a4e3-4286-b81b-9a29461ce642 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:48.411992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff60a9f-a4e3-4286-b81b-9a29461ce642 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:48.414082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fff60a9f-a4e3-4286-b81b-9a29461ce642 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:48.437730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 689/1382] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:50.735815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb08c0-15fe-450a-97ce-c3acc4426721 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:50.736606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb08c0-15fe-450a-97ce-c3acc4426721 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:50.736606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb08c0-15fe-450a-97ce-c3acc4426721 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:50.737720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 694/1383] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:05:50.743938 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce48c1-bdba-4560-a34f-13fee666a7f6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:50.744441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce48c1-bdba-4560-a34f-13fee666a7f6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:50.744696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ce48c1-bdba-4560-a34f-13fee666a7f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:50.816030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a3ce48c1-bdba-4560-a34f-13fee666a7f6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:50.817730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a3ce48c1-bdba-4560-a34f-13fee666a7f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:50.893326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:50.942989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 690/1384] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:51.229811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb08c0-15fe-450a-97ce-c3acc4426721 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:51.231379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b9cb08c0-15fe-450a-97ce-c3acc4426721 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:51.408802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb08c0-15fe-450a-97ce-c3acc4426721 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:51.470533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:51.549313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdc0a3fb-309b-4932-9aff-81f9af6c35b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:51.549313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc0a3fb-309b-4932-9aff-81f9af6c35b6 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:51.549663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc0a3fb-309b-4932-9aff-81f9af6c35b6 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:51.549990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc0a3fb-309b-4932-9aff-81f9af6c35b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:51.552057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bdc0a3fb-309b-4932-9aff-81f9af6c35b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:51.664028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc0a3fb-309b-4932-9aff-81f9af6c35b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:51.665622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdc0a3fb-309b-4932-9aff-81f9af6c35b6 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:51.684523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 695/1385] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 455 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:54.032714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3898368-caf4-4971-9d20-2a96f47d8f1f None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:54.033316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3898368-caf4-4971-9d20-2a96f47d8f1f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:54.033550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3898368-caf4-4971-9d20-2a96f47d8f1f None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:54.034862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 691/1386] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:54.040353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a0227-5d19-4c99-9d5e-fdfc5952528d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:54.040814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a0227-5d19-4c99-9d5e-fdfc5952528d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:54.040913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b60a0227-5d19-4c99-9d5e-fdfc5952528d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:54.113091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b60a0227-5d19-4c99-9d5e-fdfc5952528d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:54.114336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b60a0227-5d19-4c99-9d5e-fdfc5952528d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:54.122888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:54.218680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 696/1387] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:54.504494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3898368-caf4-4971-9d20-2a96f47d8f1f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:54.506042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a3898368-caf4-4971-9d20-2a96f47d8f1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:54.656673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3898368-caf4-4971-9d20-2a96f47d8f1f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:54.706041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:54.774694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-482ad6c5-d711-45d7-8921-e2f5ba84bb99 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:54.775215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482ad6c5-d711-45d7-8921-e2f5ba84bb99 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:54.775482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482ad6c5-d711-45d7-8921-e2f5ba84bb99 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:54.775766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-482ad6c5-d711-45d7-8921-e2f5ba84bb99 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:54.777337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-482ad6c5-d711-45d7-8921-e2f5ba84bb99 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:54.870995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482ad6c5-d711-45d7-8921-e2f5ba84bb99 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:54.872632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-482ad6c5-d711-45d7-8921-e2f5ba84bb99 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:54.892331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 692/1388] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:55.324928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03e021-0445-4810-ae29-7802d5d07c7a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:55.325482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03e021-0445-4810-ae29-7802d5d07c7a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:55.325482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee03e021-0445-4810-ae29-7802d5d07c7a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:55.326920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 697/1389] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:55.332584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c596f716-c5fc-4521-81b1-b06da0ded965 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:55.333002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c596f716-c5fc-4521-81b1-b06da0ded965 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:55.333203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c596f716-c5fc-4521-81b1-b06da0ded965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:55.399731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:55.402857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c596f716-c5fc-4521-81b1-b06da0ded965 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:55.404819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c596f716-c5fc-4521-81b1-b06da0ded965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:55.509144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 693/1390] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:55.807699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d6e5c-3b0c-440d-a53b-c5c99dc25aba None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:55.808060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d6e5c-3b0c-440d-a53b-c5c99dc25aba None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:55.808463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c00d6e5c-3b0c-440d-a53b-c5c99dc25aba None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:55.809832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 698/1391] 10.222.0.61 () {54 vars in 962 bytes} [Thu Sep 29 02:05:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:05:55.816497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc3844-ed50-447a-bea9-a0c4553af803 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:55.816791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc3844-ed50-447a-bea9-a0c4553af803 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:55.817074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bc3844-ed50-447a-bea9-a0c4553af803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:55.896320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b9bc3844-ed50-447a-bea9-a0c4553af803 None None] MFA Rules not processed for user `d6cfd5241a0147d9bfe0b7addb3e1851`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:55.897544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b9bc3844-ed50-447a-bea9-a0c4553af803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:55.970955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:55.977229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 694/1392] 10.222.0.61 () {58 vars in 1059 bytes} [Thu Sep 29 02:05:55 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:55.986582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00d6e5c-3b0c-440d-a53b-c5c99dc25aba None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:55.988094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c00d6e5c-3b0c-440d-a53b-c5c99dc25aba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:56.097873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c00d6e5c-3b0c-440d-a53b-c5c99dc25aba None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:56.138822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:56.179567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88d6aee8-1be9-478e-87f1-59c413971351 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:56.180191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d6aee8-1be9-478e-87f1-59c413971351 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:56.180460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d6aee8-1be9-478e-87f1-59c413971351 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:56.180725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88d6aee8-1be9-478e-87f1-59c413971351 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:56.182357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-88d6aee8-1be9-478e-87f1-59c413971351 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:56.301508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d6aee8-1be9-478e-87f1-59c413971351 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:56.302984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88d6aee8-1be9-478e-87f1-59c413971351 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:56.321865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 699/1393] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:05:57.103933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8456d7-de5f-4a3f-96c4-6a9c6283c62c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:57.104474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8456d7-de5f-4a3f-96c4-6a9c6283c62c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:57.104737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed8456d7-de5f-4a3f-96c4-6a9c6283c62c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:57.140945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc12fb36-a41c-417d-bcce-0d12b05e08df None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:57.141528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc12fb36-a41c-417d-bcce-0d12b05e08df None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:57.141528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc12fb36-a41c-417d-bcce-0d12b05e08df None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:57.142919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 700/1394] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:05:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:05:57.147904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:57.148181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:57.148455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:57.173800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ed8456d7-de5f-4a3f-96c4-6a9c6283c62c None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:57.175038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ed8456d7-de5f-4a3f-96c4-6a9c6283c62c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:57.221237 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:57.222468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:57.241221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:57.268172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 695/1395] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:05:57 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:57.310151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e934ca3a-f6a5-467a-96e3-2bc8b757d78a None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:57.310438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e934ca3a-f6a5-467a-96e3-2bc8b757d78a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:57.310711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e934ca3a-f6a5-467a-96e3-2bc8b757d78a None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:57.312409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 696/1396] 10.222.0.61 () {54 vars in 943 bytes} [Thu Sep 29 02:05:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:05:57.318933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:57.319215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:57.319484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:57.323943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 701/1397] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:05:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:05:57.397282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] MFA Rules not processed for user `34b928e9fe314d0fb61122b7a558c1b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:05:57.398539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:57.426516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:57.504628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 697/1398] 10.222.0.61 () {58 vars in 1040 bytes} [Thu Sep 29 02:05:57 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:05:57.514544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:57.515787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:57.612901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:05:57.614103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:57.627616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7934f932-c95a-4620-aeb5-8e2082b1f50a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:57.682927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:57.719905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24f1b8a0-3026-414f-b6bd-7d7329982dfe None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:05:57.741594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4aa694a-0093-4e36-9374-94c385f86b53 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:57.742355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aa694a-0093-4e36-9374-94c385f86b53 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:57.743145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aa694a-0093-4e36-9374-94c385f86b53 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:57.743423 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4aa694a-0093-4e36-9374-94c385f86b53 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:57.745127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b4aa694a-0093-4e36-9374-94c385f86b53 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:57.770105 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:05:57.834991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aa694a-0093-4e36-9374-94c385f86b53 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:57.836643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4aa694a-0093-4e36-9374-94c385f86b53 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:57.844717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 702/1399] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:05:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 331 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:05:57.846367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e59b9af-cb25-4f47-b7a9-7bc7d040532c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:05:57.847116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e59b9af-cb25-4f47-b7a9-7bc7d040532c service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:05:57.847320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e59b9af-cb25-4f47-b7a9-7bc7d040532c service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:05:57.847508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e59b9af-cb25-4f47-b7a9-7bc7d040532c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:05:57.849244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9e59b9af-cb25-4f47-b7a9-7bc7d040532c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:05:57.963188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e59b9af-cb25-4f47-b7a9-7bc7d040532c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:05:57.964908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e59b9af-cb25-4f47-b7a9-7bc7d040532c service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:05:57.989316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 698/1400] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:05:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 377 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:06:00.222125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0dcca4-47a1-4217-912f-04e0516ddbc7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:00.222125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0dcca4-47a1-4217-912f-04e0516ddbc7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:00.222125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0dcca4-47a1-4217-912f-04e0516ddbc7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:00.225637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 703/1401] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:06:00 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:00.227550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa87412-7f2d-4a6c-94f9-d062cb0465d8 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:00.228925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa87412-7f2d-4a6c-94f9-d062cb0465d8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:00.228925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa87412-7f2d-4a6c-94f9-d062cb0465d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:00.368170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3fa87412-7f2d-4a6c-94f9-d062cb0465d8 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:00.369421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3fa87412-7f2d-4a6c-94f9-d062cb0465d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:00.372838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:00.484756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 699/1402] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 259 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:00.782756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0dcca4-47a1-4217-912f-04e0516ddbc7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:00.784556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4b0dcca4-47a1-4217-912f-04e0516ddbc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:00.878334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:00.952489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0dcca4-47a1-4217-912f-04e0516ddbc7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:01.080220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e4d6c0-d503-4ccc-b47b-6b339144029f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:01.080840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e4d6c0-d503-4ccc-b47b-6b339144029f service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:01.081109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e4d6c0-d503-4ccc-b47b-6b339144029f service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:01.081380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e4d6c0-d503-4ccc-b47b-6b339144029f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:01.083024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-46e4d6c0-d503-4ccc-b47b-6b339144029f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:01.136235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:01.193623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e4d6c0-d503-4ccc-b47b-6b339144029f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:01.195420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e4d6c0-d503-4ccc-b47b-6b339144029f service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:01.214037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 704/1403] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 432 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:03.433128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d231a967-0bdf-4d16-81f0-92e61aa685d2 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:03.433725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d231a967-0bdf-4d16-81f0-92e61aa685d2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:03.433725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d231a967-0bdf-4d16-81f0-92e61aa685d2 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:03.435212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 700/1404] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:06:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:03.440992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe39f413-3ce4-41e0-b1ed-4f0e0c2d0d72 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:03.441733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe39f413-3ce4-41e0-b1ed-4f0e0c2d0d72 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:03.442184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe39f413-3ce4-41e0-b1ed-4f0e0c2d0d72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:03.511156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fe39f413-3ce4-41e0-b1ed-4f0e0c2d0d72 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:03.512385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fe39f413-3ce4-41e0-b1ed-4f0e0c2d0d72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:03.628565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 705/1405] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:03.918907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d231a967-0bdf-4d16-81f0-92e61aa685d2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:03.920463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d231a967-0bdf-4d16-81f0-92e61aa685d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:04.067790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:04.080960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d231a967-0bdf-4d16-81f0-92e61aa685d2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:04.198368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe2351b-f152-4b3a-9322-f1a6102fcc1d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:04.198880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2351b-f152-4b3a-9322-f1a6102fcc1d service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:04.199158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2351b-f152-4b3a-9322-f1a6102fcc1d service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:04.199421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe2351b-f152-4b3a-9322-f1a6102fcc1d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:04.201125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2fe2351b-f152-4b3a-9322-f1a6102fcc1d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:04.312665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe2351b-f152-4b3a-9322-f1a6102fcc1d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:04.314507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe2351b-f152-4b3a-9322-f1a6102fcc1d service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:04.324561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:04.337212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 701/1406] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:06.563822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12a209-add6-4dc7-8c12-98208f3e221d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:06.564483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12a209-add6-4dc7-8c12-98208f3e221d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:06.564483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc12a209-add6-4dc7-8c12-98208f3e221d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:06.565759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 706/1407] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:06:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:06.571115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfedad-5c06-4c88-8ba0-55cb048a1f9e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:06.571557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfedad-5c06-4c88-8ba0-55cb048a1f9e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:06.571648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4cfedad-5c06-4c88-8ba0-55cb048a1f9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:06.641804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a4cfedad-5c06-4c88-8ba0-55cb048a1f9e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:06.643268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a4cfedad-5c06-4c88-8ba0-55cb048a1f9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:06.766157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 702/1408] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:06:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:07.058656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc12a209-add6-4dc7-8c12-98208f3e221d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:07.060208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fc12a209-add6-4dc7-8c12-98208f3e221d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:07.083258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:07.220798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc12a209-add6-4dc7-8c12-98208f3e221d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:07.321680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:07.346478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-660b97ad-33ba-4b02-b7cf-db5d7ea0c413 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:07.347084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b97ad-33ba-4b02-b7cf-db5d7ea0c413 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:07.347372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b97ad-33ba-4b02-b7cf-db5d7ea0c413 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:07.347619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-660b97ad-33ba-4b02-b7cf-db5d7ea0c413 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:07.349206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-660b97ad-33ba-4b02-b7cf-db5d7ea0c413 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:07.448574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660b97ad-33ba-4b02-b7cf-db5d7ea0c413 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:07.450220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-660b97ad-33ba-4b02-b7cf-db5d7ea0c413 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:07.471102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 707/1409] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:16.757148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=68 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:17.468587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=54 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:20.486564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fda345-a008-4154-8aa5-1840e6542f4f None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:20.487175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fda345-a008-4154-8aa5-1840e6542f4f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:20.487647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2fda345-a008-4154-8aa5-1840e6542f4f None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:20.489023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 703/1410] 10.222.0.61 () {54 vars in 918 bytes} [Thu Sep 29 02:06:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:20.496395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba51669-3d77-4d7a-8277-922128b87b95 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:20.496828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba51669-3d77-4d7a-8277-922128b87b95 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:20.496934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dba51669-3d77-4d7a-8277-922128b87b95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:20.579458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-dba51669-3d77-4d7a-8277-922128b87b95 None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:20.580745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-dba51669-3d77-4d7a-8277-922128b87b95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:20.697258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 708/1411] 10.222.0.61 () {58 vars in 1015 bytes} [Thu Sep 29 02:06:20 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:20.713504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740e13b-a14e-424d-949e-59bdb4f734fe None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:20.713813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740e13b-a14e-424d-949e-59bdb4f734fe None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:20.714220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2740e13b-a14e-424d-949e-59bdb4f734fe None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:20.716220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 704/1412] 10.222.0.61 () {54 vars in 918 bytes} [Thu Sep 29 02:06:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:20.721373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb193ee-c449-470f-8477-118a9a214e00 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:20.721627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb193ee-c449-470f-8477-118a9a214e00 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:20.721899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb193ee-c449-470f-8477-118a9a214e00 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:20.801625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:20.804987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4cb193ee-c449-470f-8477-118a9a214e00 None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:20.807437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4cb193ee-c449-470f-8477-118a9a214e00 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:20.942936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 709/1413] 10.222.0.61 () {58 vars in 1015 bytes} [Thu Sep 29 02:06:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:06:23.521593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da44d88c-3992-4c00-8f63-9cfdb94198cd None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:23.522563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da44d88c-3992-4c00-8f63-9cfdb94198cd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:23.522563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da44d88c-3992-4c00-8f63-9cfdb94198cd None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:23.524595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 705/1414] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:06:23 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:23.531336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b75f08b-7e7a-463a-b035-f777bae4d457 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:23.532028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b75f08b-7e7a-463a-b035-f777bae4d457 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:23.532119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b75f08b-7e7a-463a-b035-f777bae4d457 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:23.619274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3b75f08b-7e7a-463a-b035-f777bae4d457 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:23.620496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3b75f08b-7e7a-463a-b035-f777bae4d457 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:23.681745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:23.727846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 710/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:06:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:24.018423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da44d88c-3992-4c00-8f63-9cfdb94198cd None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:24.020564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-da44d88c-3992-4c00-8f63-9cfdb94198cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:24.194765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da44d88c-3992-4c00-8f63-9cfdb94198cd None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:24.277398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:24.323643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d42d07-76bc-4364-9d75-1fa9c6c8e9d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:24.324260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d42d07-76bc-4364-9d75-1fa9c6c8e9d1 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:24.324538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d42d07-76bc-4364-9d75-1fa9c6c8e9d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:24.324832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d42d07-76bc-4364-9d75-1fa9c6c8e9d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:24.326502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-72d42d07-76bc-4364-9d75-1fa9c6c8e9d1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:24.429078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d42d07-76bc-4364-9d75-1fa9c6c8e9d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:24.430751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d42d07-76bc-4364-9d75-1fa9c6c8e9d1 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:24.449116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 706/1416] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:25.587622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d01aa-b79e-4947-8800-9eeed897ac88 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:25.588420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d01aa-b79e-4947-8800-9eeed897ac88 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:25.588420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d9d01aa-b79e-4947-8800-9eeed897ac88 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:25.590490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 711/1417] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:25.597466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47871e-6e50-4084-9725-ab7f37902611 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:25.597943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47871e-6e50-4084-9725-ab7f37902611 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:25.598083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb47871e-6e50-4084-9725-ab7f37902611 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:25.674913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-bb47871e-6e50-4084-9725-ab7f37902611 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:25.676119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bb47871e-6e50-4084-9725-ab7f37902611 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:25.703604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:25.778664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 707/1418] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:25.934488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9d01aa-b79e-4947-8800-9eeed897ac88 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:25.937023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6d9d01aa-b79e-4947-8800-9eeed897ac88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:26.100328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d9d01aa-b79e-4947-8800-9eeed897ac88 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:26.154823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:26.231244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f0aa2d-28e2-4c36-bbfb-bd20a846e914 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:26.231748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0aa2d-28e2-4c36-bbfb-bd20a846e914 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:26.232018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0aa2d-28e2-4c36-bbfb-bd20a846e914 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:26.232284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0aa2d-28e2-4c36-bbfb-bd20a846e914 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:26.233871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f9f0aa2d-28e2-4c36-bbfb-bd20a846e914 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:26.335048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f0aa2d-28e2-4c36-bbfb-bd20a846e914 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:26.336647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f0aa2d-28e2-4c36-bbfb-bd20a846e914 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:26.355347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 712/1419] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 422 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:27.484417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d6393c-c61f-46ed-a295-1845f6319618 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:27.485280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d6393c-c61f-46ed-a295-1845f6319618 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:27.485280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d6393c-c61f-46ed-a295-1845f6319618 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:27.487349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 708/1420] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:27.495365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d19fa6-8a8c-4cfd-808d-4f375b02f08c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:27.495865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d19fa6-8a8c-4cfd-808d-4f375b02f08c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:27.495865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78d19fa6-8a8c-4cfd-808d-4f375b02f08c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:27.556645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:27.583838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-78d19fa6-8a8c-4cfd-808d-4f375b02f08c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:27.585076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-78d19fa6-8a8c-4cfd-808d-4f375b02f08c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:27.695825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 713/1421] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:27.849356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d6393c-c61f-46ed-a295-1845f6319618 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:27.851831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b3d6393c-c61f-46ed-a295-1845f6319618 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:28.011385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d6393c-c61f-46ed-a295-1845f6319618 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:28.024054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:28.139151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cc7afc9-806e-4850-8ab5-2bf13b5aaf48 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:28.139664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7afc9-806e-4850-8ab5-2bf13b5aaf48 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:28.139940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7afc9-806e-4850-8ab5-2bf13b5aaf48 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:28.140205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cc7afc9-806e-4850-8ab5-2bf13b5aaf48 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:28.141803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8cc7afc9-806e-4850-8ab5-2bf13b5aaf48 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:28.236916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc7afc9-806e-4850-8ab5-2bf13b5aaf48 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:28.238613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cc7afc9-806e-4850-8ab5-2bf13b5aaf48 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:28.256877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 709/1422] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:06:29.378786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7af967-809b-4244-84e7-428aa44f6f0a None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:29.379561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7af967-809b-4244-84e7-428aa44f6f0a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:29.380021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7af967-809b-4244-84e7-428aa44f6f0a None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:29.382641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 714/1423] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:29 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:29.389541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca147c7-eba1-4b9e-9b3c-73798821b3fe None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:29.390290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca147c7-eba1-4b9e-9b3c-73798821b3fe None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:29.390429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ca147c7-eba1-4b9e-9b3c-73798821b3fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:29.413022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:29.475418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6ca147c7-eba1-4b9e-9b3c-73798821b3fe None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:29.476625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6ca147c7-eba1-4b9e-9b3c-73798821b3fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:29.578548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 710/1424] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:29.728879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7af967-809b-4244-84e7-428aa44f6f0a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:29.731201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-dc7af967-809b-4244-84e7-428aa44f6f0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:29.859788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:29.896156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7af967-809b-4244-84e7-428aa44f6f0a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:30.015324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b08da46-a047-490f-bdb8-0643e83c5a8f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:30.016029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b08da46-a047-490f-bdb8-0643e83c5a8f service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:30.016379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b08da46-a047-490f-bdb8-0643e83c5a8f service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:30.016721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b08da46-a047-490f-bdb8-0643e83c5a8f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:30.019228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9b08da46-a047-490f-bdb8-0643e83c5a8f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:30.104984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:30.120516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b08da46-a047-490f-bdb8-0643e83c5a8f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:30.122166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b08da46-a047-490f-bdb8-0643e83c5a8f service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:30.141233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 715/1425] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 414 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:31.239990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9f468-0017-439c-971a-cce7f6321341 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:31.240554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9f468-0017-439c-971a-cce7f6321341 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:31.240554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b9f468-0017-439c-971a-cce7f6321341 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:31.241855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 711/1426] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:31 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:31.251360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0207a320-7831-4d83-b0c9-270b3e7549e0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:31.251822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0207a320-7831-4d83-b0c9-270b3e7549e0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:31.251924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0207a320-7831-4d83-b0c9-270b3e7549e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:31.334095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0207a320-7831-4d83-b0c9-270b3e7549e0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:31.335350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0207a320-7831-4d83-b0c9-270b3e7549e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:31.444287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 716/1427] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:31.594986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9f468-0017-439c-971a-cce7f6321341 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:31.596931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c4b9f468-0017-439c-971a-cce7f6321341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:31.684630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:31.750658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b9f468-0017-439c-971a-cce7f6321341 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:31.870630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112dace1-29eb-4923-8358-5d38eff92bdc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:31.871245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112dace1-29eb-4923-8358-5d38eff92bdc service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:31.871516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112dace1-29eb-4923-8358-5d38eff92bdc service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:31.871784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112dace1-29eb-4923-8358-5d38eff92bdc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:31.873395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-112dace1-29eb-4923-8358-5d38eff92bdc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:31.922448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:31.966125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112dace1-29eb-4923-8358-5d38eff92bdc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:31.967748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112dace1-29eb-4923-8358-5d38eff92bdc service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:31.986101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 712/1428] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 392 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:33.055011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7343b391-e5a6-4035-a40c-0c4817a328a9 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:33.055932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7343b391-e5a6-4035-a40c-0c4817a328a9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:33.055932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7343b391-e5a6-4035-a40c-0c4817a328a9 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:33.058144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 717/1429] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:33 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:33.067925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8a2ab4-2fe0-46d7-b73e-14b5d4ec24e7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:33.068425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8a2ab4-2fe0-46d7-b73e-14b5d4ec24e7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:33.068425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc8a2ab4-2fe0-46d7-b73e-14b5d4ec24e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:33.155113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cc8a2ab4-2fe0-46d7-b73e-14b5d4ec24e7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:33.156371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cc8a2ab4-2fe0-46d7-b73e-14b5d4ec24e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:33.257728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 713/1430] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:33.409950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7343b391-e5a6-4035-a40c-0c4817a328a9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:33.412589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7343b391-e5a6-4035-a40c-0c4817a328a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:33.475231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:33.578835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7343b391-e5a6-4035-a40c-0c4817a328a9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:33.696655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-341eb6da-a4bc-4c4d-b7d8-c9a88a4fae23 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:33.697177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341eb6da-a4bc-4c4d-b7d8-c9a88a4fae23 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:33.697448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341eb6da-a4bc-4c4d-b7d8-c9a88a4fae23 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:33.697714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-341eb6da-a4bc-4c4d-b7d8-c9a88a4fae23 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:33.699350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-341eb6da-a4bc-4c4d-b7d8-c9a88a4fae23 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:33.706669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:33.803417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341eb6da-a4bc-4c4d-b7d8-c9a88a4fae23 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:33.805032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-341eb6da-a4bc-4c4d-b7d8-c9a88a4fae23 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:33.824561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 718/1431] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 416 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:34.882576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348cd9b6-a11a-4c1e-b4fd-8ff38ea625dc None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:34.883175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348cd9b6-a11a-4c1e-b4fd-8ff38ea625dc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:34.883175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348cd9b6-a11a-4c1e-b4fd-8ff38ea625dc None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:34.885250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 714/1432] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:34 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:34.895760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d85e99-0874-4599-beb7-574f7ff85266 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:34.896209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d85e99-0874-4599-beb7-574f7ff85266 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:34.896311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d85e99-0874-4599-beb7-574f7ff85266 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:34.978849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-89d85e99-0874-4599-beb7-574f7ff85266 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:34.980132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-89d85e99-0874-4599-beb7-574f7ff85266 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:35.053695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:35.088845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 719/1433] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:35.242797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348cd9b6-a11a-4c1e-b4fd-8ff38ea625dc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:35.245018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-348cd9b6-a11a-4c1e-b4fd-8ff38ea625dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:35.267103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:35.407917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348cd9b6-a11a-4c1e-b4fd-8ff38ea625dc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:35.501587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:35.527951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e1a92d-6ba5-41bd-a0f7-b8ab68bcab43 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:35.528561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1a92d-6ba5-41bd-a0f7-b8ab68bcab43 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:35.528826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1a92d-6ba5-41bd-a0f7-b8ab68bcab43 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:35.529098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e1a92d-6ba5-41bd-a0f7-b8ab68bcab43 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:35.530731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e2e1a92d-6ba5-41bd-a0f7-b8ab68bcab43 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:35.630546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e1a92d-6ba5-41bd-a0f7-b8ab68bcab43 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:35.632190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e1a92d-6ba5-41bd-a0f7-b8ab68bcab43 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:35.654551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 715/1434] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 413 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:36.722357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d4c24e-ff8a-4a92-ace2-eb37624f5f63 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:36.722951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d4c24e-ff8a-4a92-ace2-eb37624f5f63 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:36.722951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54d4c24e-ff8a-4a92-ace2-eb37624f5f63 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:36.724382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 720/1435] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:36.731719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e9a5f-1e0d-462a-a1d7-18ba3a63156f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:36.732198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e9a5f-1e0d-462a-a1d7-18ba3a63156f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:36.732305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-790e9a5f-1e0d-462a-a1d7-18ba3a63156f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:36.824277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-790e9a5f-1e0d-462a-a1d7-18ba3a63156f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:36.825466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-790e9a5f-1e0d-462a-a1d7-18ba3a63156f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:36.870948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:36.928021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 716/1436] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 198 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:37.087793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d4c24e-ff8a-4a92-ace2-eb37624f5f63 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:37.089290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-54d4c24e-ff8a-4a92-ace2-eb37624f5f63 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:37.241710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54d4c24e-ff8a-4a92-ace2-eb37624f5f63 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:37.304324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:37.366100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c482c74c-a289-44e9-a1e6-ab2fccdb4a73 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:37.366655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482c74c-a289-44e9-a1e6-ab2fccdb4a73 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:37.366928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482c74c-a289-44e9-a1e6-ab2fccdb4a73 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:37.367193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482c74c-a289-44e9-a1e6-ab2fccdb4a73 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:37.368805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c482c74c-a289-44e9-a1e6-ab2fccdb4a73 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:37.467738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482c74c-a289-44e9-a1e6-ab2fccdb4a73 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:37.469345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482c74c-a289-44e9-a1e6-ab2fccdb4a73 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:37.487969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 721/1437] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:38.561468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce93fb1-d791-4575-b2db-47ccaf5e4d39 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:38.562057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce93fb1-d791-4575-b2db-47ccaf5e4d39 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:38.562057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce93fb1-d791-4575-b2db-47ccaf5e4d39 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:38.564405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 717/1438] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:06:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:38.573416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d25118-3229-47bf-a3ec-0497244704ee None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:38.573935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d25118-3229-47bf-a3ec-0497244704ee None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:38.573935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d25118-3229-47bf-a3ec-0497244704ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:38.655487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-64d25118-3229-47bf-a3ec-0497244704ee None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:38.656702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-64d25118-3229-47bf-a3ec-0497244704ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:38.658763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:38.761608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 722/1439] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:06:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:06:38.915855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce93fb1-d791-4575-b2db-47ccaf5e4d39 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:38.918223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7ce93fb1-d791-4575-b2db-47ccaf5e4d39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:39.081500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce93fb1-d791-4575-b2db-47ccaf5e4d39 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:39.110834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:39.210417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6443e8c1-9865-43bf-bfa0-c2b7a8db1cda service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:39.210932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6443e8c1-9865-43bf-bfa0-c2b7a8db1cda service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:39.211208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6443e8c1-9865-43bf-bfa0-c2b7a8db1cda service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:39.211478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6443e8c1-9865-43bf-bfa0-c2b7a8db1cda service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:39.213101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6443e8c1-9865-43bf-bfa0-c2b7a8db1cda service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:39.316971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6443e8c1-9865-43bf-bfa0-c2b7a8db1cda service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:39.318762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6443e8c1-9865-43bf-bfa0-c2b7a8db1cda service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:39.337629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 718/1440] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:41.424639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1429c91-16bb-4f56-b1d7-e68234d3bba2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:41.425217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1429c91-16bb-4f56-b1d7-e68234d3bba2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:41.425217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1429c91-16bb-4f56-b1d7-e68234d3bba2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:41.426602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 723/1441] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:06:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:41.433181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc2587-f24c-4c67-8983-a4c90db26155 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:41.434022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc2587-f24c-4c67-8983-a4c90db26155 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:41.434022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cc2587-f24c-4c67-8983-a4c90db26155 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:41.484898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:41.510910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-70cc2587-f24c-4c67-8983-a4c90db26155 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:41.512148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-70cc2587-f24c-4c67-8983-a4c90db26155 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:41.614534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 719/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:06:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:41.905592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1429c91-16bb-4f56-b1d7-e68234d3bba2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:41.908128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e1429c91-16bb-4f56-b1d7-e68234d3bba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:42.016338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1429c91-16bb-4f56-b1d7-e68234d3bba2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:42.051299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:42.097358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deca4cbc-3098-4cdb-99af-4839d6f30551 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:42.097962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca4cbc-3098-4cdb-99af-4839d6f30551 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:42.098267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca4cbc-3098-4cdb-99af-4839d6f30551 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:42.098539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deca4cbc-3098-4cdb-99af-4839d6f30551 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:42.100334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-deca4cbc-3098-4cdb-99af-4839d6f30551 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:42.219214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deca4cbc-3098-4cdb-99af-4839d6f30551 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:42.220582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-deca4cbc-3098-4cdb-99af-4839d6f30551 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:42.242490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 724/1443] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:44.565699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe8a5e-f4aa-4774-afc3-856e049ece9b None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:44.566240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe8a5e-f4aa-4774-afc3-856e049ece9b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:44.566240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe8a5e-f4aa-4774-afc3-856e049ece9b None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:44.568234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 720/1444] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:06:44 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:44.576016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8028577-fc17-408c-83c2-38d6aa3acdeb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:44.576445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8028577-fc17-408c-83c2-38d6aa3acdeb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:44.576541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8028577-fc17-408c-83c2-38d6aa3acdeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:44.651786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b8028577-fc17-408c-83c2-38d6aa3acdeb None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:44.653019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b8028577-fc17-408c-83c2-38d6aa3acdeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:44.666812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:44.759060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 725/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:06:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:45.044512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe8a5e-f4aa-4774-afc3-856e049ece9b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:45.046140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7dfe8a5e-f4aa-4774-afc3-856e049ece9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:45.153067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe8a5e-f4aa-4774-afc3-856e049ece9b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:45.166153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:45.234355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-934f4b53-fb1a-4df9-9143-135ee6a2930b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:45.234966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f4b53-fb1a-4df9-9143-135ee6a2930b service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:45.235241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f4b53-fb1a-4df9-9143-135ee6a2930b service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:45.235510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-934f4b53-fb1a-4df9-9143-135ee6a2930b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:45.237118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-934f4b53-fb1a-4df9-9143-135ee6a2930b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:45.374422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934f4b53-fb1a-4df9-9143-135ee6a2930b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:45.375884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-934f4b53-fb1a-4df9-9143-135ee6a2930b service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:45.395873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 721/1446] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:47.970826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba7bb95-c9c6-4d35-8234-a30b640da2fe None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:47.971435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba7bb95-c9c6-4d35-8234-a30b640da2fe None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:47.971435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba7bb95-c9c6-4d35-8234-a30b640da2fe None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:47.972732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 726/1447] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:06:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:47.979014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04779f73-213f-4637-83a2-72048b4c25ee None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:47.979473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04779f73-213f-4637-83a2-72048b4c25ee None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:47.979603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04779f73-213f-4637-83a2-72048b4c25ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:48.028954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:48.050972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-04779f73-213f-4637-83a2-72048b4c25ee None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:48.052171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-04779f73-213f-4637-83a2-72048b4c25ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:48.152110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 722/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:06:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:48.435676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba7bb95-c9c6-4d35-8234-a30b640da2fe None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:48.438182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cba7bb95-c9c6-4d35-8234-a30b640da2fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:48.567303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba7bb95-c9c6-4d35-8234-a30b640da2fe None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:48.612120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:48.651612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac51c346-651f-40b7-9090-ef63700ce4b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:48.652233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac51c346-651f-40b7-9090-ef63700ce4b0 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:48.652506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac51c346-651f-40b7-9090-ef63700ce4b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:48.652777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac51c346-651f-40b7-9090-ef63700ce4b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:48.654488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ac51c346-651f-40b7-9090-ef63700ce4b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:48.773086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac51c346-651f-40b7-9090-ef63700ce4b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:48.774651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac51c346-651f-40b7-9090-ef63700ce4b0 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:48.793430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 727/1449] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 359 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:51.103734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c7dbfd-78f4-40e2-8277-bad3a53eca41 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:51.104995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c7dbfd-78f4-40e2-8277-bad3a53eca41 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:51.104995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c7dbfd-78f4-40e2-8277-bad3a53eca41 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:51.107602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 723/1450] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:06:51 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:51.114087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8492c3-2627-48e4-b551-3c230fa814dc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:51.114508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8492c3-2627-48e4-b551-3c230fa814dc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:51.114625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e8492c3-2627-48e4-b551-3c230fa814dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:51.186495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2e8492c3-2627-48e4-b551-3c230fa814dc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:51.187723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2e8492c3-2627-48e4-b551-3c230fa814dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:51.209860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:51.295236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 728/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:06:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:51.577541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c7dbfd-78f4-40e2-8277-bad3a53eca41 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:51.580000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a5c7dbfd-78f4-40e2-8277-bad3a53eca41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:51.695779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c7dbfd-78f4-40e2-8277-bad3a53eca41 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:51.718057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:51.772622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79ed7e5d-6983-41bf-b804-041d96552ed1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:51.773232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed7e5d-6983-41bf-b804-041d96552ed1 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:51.773505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed7e5d-6983-41bf-b804-041d96552ed1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:51.773771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79ed7e5d-6983-41bf-b804-041d96552ed1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:51.775414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-79ed7e5d-6983-41bf-b804-041d96552ed1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:51.901524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ed7e5d-6983-41bf-b804-041d96552ed1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:51.903008 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79ed7e5d-6983-41bf-b804-041d96552ed1 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:51.922329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 724/1452] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:06:54.132704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c351224-d7c7-4905-a870-7d833bede5de None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:54.133329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c351224-d7c7-4905-a870-7d833bede5de None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:54.133329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c351224-d7c7-4905-a870-7d833bede5de None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:54.134627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 729/1453] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:06:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:54.141198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21225e1e-0757-417f-b4ab-c813a3151baf None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:54.141905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21225e1e-0757-417f-b4ab-c813a3151baf None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:54.142128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21225e1e-0757-417f-b4ab-c813a3151baf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:54.219833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:54.222608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-21225e1e-0757-417f-b4ab-c813a3151baf None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:54.223897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-21225e1e-0757-417f-b4ab-c813a3151baf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:54.324686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 725/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:06:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:54.605722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c351224-d7c7-4905-a870-7d833bede5de None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:54.607282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8c351224-d7c7-4905-a870-7d833bede5de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:54.722865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c351224-d7c7-4905-a870-7d833bede5de None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:54.775175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:54.802271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5ab4a27-21eb-4772-a4bc-c1c52cfd9821 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:54.802808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab4a27-21eb-4772-a4bc-c1c52cfd9821 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:54.803088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab4a27-21eb-4772-a4bc-c1c52cfd9821 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:54.803371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5ab4a27-21eb-4772-a4bc-c1c52cfd9821 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:54.805045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f5ab4a27-21eb-4772-a4bc-c1c52cfd9821 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:54.917087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ab4a27-21eb-4772-a4bc-c1c52cfd9821 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:54.918634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5ab4a27-21eb-4772-a4bc-c1c52cfd9821 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:54.938342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 730/1455] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:06:57.132369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb9ea5-77e2-4247-bf95-ef664f320a87 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:57.132969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb9ea5-77e2-4247-bf95-ef664f320a87 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:57.132969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb9ea5-77e2-4247-bf95-ef664f320a87 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:57.135046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 726/1456] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:06:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:06:57.141455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612a40e-1234-4428-9310-c972d170891f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:57.141930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612a40e-1234-4428-9310-c972d170891f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:57.142066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6612a40e-1234-4428-9310-c972d170891f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:57.213194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6612a40e-1234-4428-9310-c972d170891f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:06:57.214493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6612a40e-1234-4428-9310-c972d170891f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:57.247182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:57.318378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 731/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:06:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:06:57.609934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb9ea5-77e2-4247-bf95-ef664f320a87 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:06:57.612462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9ebb9ea5-77e2-4247-bf95-ef664f320a87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:57.731767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb9ea5-77e2-4247-bf95-ef664f320a87 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:06:57.760855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:06:57.808224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71b012f-6918-4822-8946-9d833f292d62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:06:57.808853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71b012f-6918-4822-8946-9d833f292d62 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:06:57.809123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71b012f-6918-4822-8946-9d833f292d62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:06:57.809389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71b012f-6918-4822-8946-9d833f292d62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:06:57.811004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d71b012f-6918-4822-8946-9d833f292d62 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:06:57.923375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71b012f-6918-4822-8946-9d833f292d62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:06:57.924725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d71b012f-6918-4822-8946-9d833f292d62 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:06:57.943115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 727/1458] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:06:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 334 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:00.213273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d4d6c-f056-4b07-a6ce-823bc87d2e73 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:00.214144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d4d6c-f056-4b07-a6ce-823bc87d2e73 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:00.214144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512d4d6c-f056-4b07-a6ce-823bc87d2e73 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:00.216486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 732/1459] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:07:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:00.222011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d79f05-227b-455f-9155-b14f6c597787 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:00.222454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d79f05-227b-455f-9155-b14f6c597787 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:00.222549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6d79f05-227b-455f-9155-b14f6c597787 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:00.300147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-e6d79f05-227b-455f-9155-b14f6c597787 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:00.301728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e6d79f05-227b-455f-9155-b14f6c597787 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:00.310659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:00.401404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 728/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:07:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:00.695366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d4d6c-f056-4b07-a6ce-823bc87d2e73 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:00.697666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-512d4d6c-f056-4b07-a6ce-823bc87d2e73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:00.816765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512d4d6c-f056-4b07-a6ce-823bc87d2e73 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:00.879769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:00.899140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f764ee94-b996-40c0-93b6-6296f55f2deb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:00.899748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f764ee94-b996-40c0-93b6-6296f55f2deb service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:00.900023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f764ee94-b996-40c0-93b6-6296f55f2deb service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:00.900285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f764ee94-b996-40c0-93b6-6296f55f2deb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:00.901875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f764ee94-b996-40c0-93b6-6296f55f2deb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:01.019874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f764ee94-b996-40c0-93b6-6296f55f2deb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:01.021410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f764ee94-b996-40c0-93b6-6296f55f2deb service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:01.045949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 733/1461] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:03.277378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e897a4-899c-487b-b335-905d86a28164 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:03.278047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e897a4-899c-487b-b335-905d86a28164 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:03.278047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e897a4-899c-487b-b335-905d86a28164 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:03.279299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 729/1462] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:07:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:03.285971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693b076d-7846-4b06-bed7-f7736d9288c5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:03.286800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693b076d-7846-4b06-bed7-f7736d9288c5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:03.286986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693b076d-7846-4b06-bed7-f7736d9288c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:03.370975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-693b076d-7846-4b06-bed7-f7736d9288c5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:03.372206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-693b076d-7846-4b06-bed7-f7736d9288c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:03.419282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:03.479663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 734/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:07:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:03.771327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e897a4-899c-487b-b335-905d86a28164 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:03.773830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f5e897a4-899c-487b-b335-905d86a28164 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:03.888208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e897a4-899c-487b-b335-905d86a28164 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:03.932630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:03.972187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea0af782-0646-453e-8658-1da0f27eff21 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:03.973210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0af782-0646-453e-8658-1da0f27eff21 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:03.973210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0af782-0646-453e-8658-1da0f27eff21 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:03.973424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea0af782-0646-453e-8658-1da0f27eff21 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:03.974947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ea0af782-0646-453e-8658-1da0f27eff21 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:04.098030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0af782-0646-453e-8658-1da0f27eff21 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:04.099465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea0af782-0646-453e-8658-1da0f27eff21 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:04.118874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 730/1464] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:06.328922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06db2a5-5101-46fe-bc85-83f3d7903748 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:06.329397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06db2a5-5101-46fe-bc85-83f3d7903748 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:06.329509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06db2a5-5101-46fe-bc85-83f3d7903748 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:06.330899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 735/1465] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:07:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:06.336631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9798398f-5501-4ce4-929c-9c11115f30fa None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:06.337098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9798398f-5501-4ce4-929c-9c11115f30fa None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:06.337207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9798398f-5501-4ce4-929c-9c11115f30fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:06.409303 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9798398f-5501-4ce4-929c-9c11115f30fa None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:06.410547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9798398f-5501-4ce4-929c-9c11115f30fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:06.429138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:06.647345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 731/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:07:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:06.928137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06db2a5-5101-46fe-bc85-83f3d7903748 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:06.929761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d06db2a5-5101-46fe-bc85-83f3d7903748 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:07.038499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06db2a5-5101-46fe-bc85-83f3d7903748 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:07.111521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:07.119519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dfbaf38-2bc1-4af0-a137-d17deb5f54f6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:07.120126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfbaf38-2bc1-4af0-a137-d17deb5f54f6 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:07.120396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfbaf38-2bc1-4af0-a137-d17deb5f54f6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:07.120663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dfbaf38-2bc1-4af0-a137-d17deb5f54f6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:07.122349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4dfbaf38-2bc1-4af0-a137-d17deb5f54f6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:07.251278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfbaf38-2bc1-4af0-a137-d17deb5f54f6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:07.252630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dfbaf38-2bc1-4af0-a137-d17deb5f54f6 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:07.393263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 736/1467] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:09.611508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49989890-2082-4caa-afb7-e68e4322e4c8 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:09.612111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49989890-2082-4caa-afb7-e68e4322e4c8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:09.612111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49989890-2082-4caa-afb7-e68e4322e4c8 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:09.613383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 732/1468] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:07:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:09.624028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb80bddd-efc4-4673-82c0-4adf177fe31f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:09.624560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb80bddd-efc4-4673-82c0-4adf177fe31f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:09.624560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb80bddd-efc4-4673-82c0-4adf177fe31f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:09.700297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-cb80bddd-efc4-4673-82c0-4adf177fe31f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:09.701534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cb80bddd-efc4-4673-82c0-4adf177fe31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:09.754564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:09.804444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 737/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:07:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:10.092372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49989890-2082-4caa-afb7-e68e4322e4c8 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:10.093908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-49989890-2082-4caa-afb7-e68e4322e4c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:10.213404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49989890-2082-4caa-afb7-e68e4322e4c8 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:10.276226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:10.304943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c297f4b-03f9-4d53-a17f-95eef82be430 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:10.305554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c297f4b-03f9-4d53-a17f-95eef82be430 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:10.305850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c297f4b-03f9-4d53-a17f-95eef82be430 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:10.306150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c297f4b-03f9-4d53-a17f-95eef82be430 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:10.309487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5c297f4b-03f9-4d53-a17f-95eef82be430 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:10.452551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c297f4b-03f9-4d53-a17f-95eef82be430 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:10.454794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c297f4b-03f9-4d53-a17f-95eef82be430 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:10.480234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 733/1470] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 388 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:19.802571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=22 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:20.476556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=66 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:30.302168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af7d2f-a7dd-4307-b51a-7a3a8e5647fe None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:30.303032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af7d2f-a7dd-4307-b51a-7a3a8e5647fe None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:30.303032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1af7d2f-a7dd-4307-b51a-7a3a8e5647fe None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:30.305681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 738/1471] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:07:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:30.312239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7f9af5-4bdb-44ee-baed-fc9e78786cfd None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:30.312971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7f9af5-4bdb-44ee-baed-fc9e78786cfd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:30.313085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7f9af5-4bdb-44ee-baed-fc9e78786cfd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:30.399624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1c7f9af5-4bdb-44ee-baed-fc9e78786cfd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:30.400853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1c7f9af5-4bdb-44ee-baed-fc9e78786cfd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:30.502020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 734/1472] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:07:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:30.824425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378059d-98b1-416d-9400-a93b506cb698 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:30.824718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378059d-98b1-416d-9400-a93b506cb698 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:30.824988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e378059d-98b1-416d-9400-a93b506cb698 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:30.827024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 739/1473] 10.222.0.61 () {54 vars in 962 bytes} [Thu Sep 29 02:07:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:30.832131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b13ad5-14fa-48d8-8613-d51d53f5a87e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:30.832429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b13ad5-14fa-48d8-8613-d51d53f5a87e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:30.832678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b13ad5-14fa-48d8-8613-d51d53f5a87e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:30.910949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-14b13ad5-14fa-48d8-8613-d51d53f5a87e None None] MFA Rules not processed for user `d6cfd5241a0147d9bfe0b7addb3e1851`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:30.912141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-14b13ad5-14fa-48d8-8613-d51d53f5a87e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:30.932791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:30.988993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 735/1474] 10.222.0.61 () {58 vars in 1059 bytes} [Thu Sep 29 02:07:30 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:30.997821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378059d-98b1-416d-9400-a93b506cb698 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:31.000245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e378059d-98b1-416d-9400-a93b506cb698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:31.115993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e378059d-98b1-416d-9400-a93b506cb698 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:31.197950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4877ec-9e55-47e2-9ada-bf2e4ced83c6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:31.198603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4877ec-9e55-47e2-9ada-bf2e4ced83c6 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:31.198883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4877ec-9e55-47e2-9ada-bf2e4ced83c6 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:31.199155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4877ec-9e55-47e2-9ada-bf2e4ced83c6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:31.200777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cb4877ec-9e55-47e2-9ada-bf2e4ced83c6 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:31.267650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:31.328624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4877ec-9e55-47e2-9ada-bf2e4ced83c6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:31.330038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4877ec-9e55-47e2-9ada-bf2e4ced83c6 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:31.349248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 740/1475] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:33.654870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0004d874-4fbb-4824-819e-853641d97b11 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:33.655822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0004d874-4fbb-4824-819e-853641d97b11 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:33.655989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0004d874-4fbb-4824-819e-853641d97b11 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:33.658572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 736/1476] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:07:33 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:33.666702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7591f8-430c-4d8d-9d27-3030bab635a7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:33.667189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7591f8-430c-4d8d-9d27-3030bab635a7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:33.667189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7591f8-430c-4d8d-9d27-3030bab635a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:33.737049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bb7591f8-430c-4d8d-9d27-3030bab635a7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:33.738315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bb7591f8-430c-4d8d-9d27-3030bab635a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:33.850725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 741/1477] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:07:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:07:34.122110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0004d874-4fbb-4824-819e-853641d97b11 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:34.124539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0004d874-4fbb-4824-819e-853641d97b11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:34.268919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0004d874-4fbb-4824-819e-853641d97b11 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:34.319410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:34.364905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e13312a-5af6-4f68-bc35-8a784be7f686 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:34.365507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13312a-5af6-4f68-bc35-8a784be7f686 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:34.365771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13312a-5af6-4f68-bc35-8a784be7f686 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:34.366014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e13312a-5af6-4f68-bc35-8a784be7f686 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:34.366675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13312a-5af6-4f68-bc35-8a784be7f686 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:34.368604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-4e13312a-5af6-4f68-bc35-8a784be7f686 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:07:34.368604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:07:34.368604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:07:34.369738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e13312a-5af6-4f68-bc35-8a784be7f686 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:34.390096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 737/1478] 10.222.0.61 () {56 vars in 1108 bytes} [Thu Sep 29 02:07:34 2022] GET /identity/v3/projects => generated 2553 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:07:36.544211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18a6157-c1fa-452e-9116-deaeb697f6f7 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:36.544799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18a6157-c1fa-452e-9116-deaeb697f6f7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:36.544799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e18a6157-c1fa-452e-9116-deaeb697f6f7 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:36.546151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 742/1479] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:07:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:36.551930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b870f1f-02db-4161-b0b0-c4be0e94f243 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:36.552399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b870f1f-02db-4161-b0b0-c4be0e94f243 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:36.552506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b870f1f-02db-4161-b0b0-c4be0e94f243 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:36.620300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3b870f1f-02db-4161-b0b0-c4be0e94f243 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:36.621396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3b870f1f-02db-4161-b0b0-c4be0e94f243 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:36.721110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 738/1480] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:07:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:37.017665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18a6157-c1fa-452e-9116-deaeb697f6f7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:37.019241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e18a6157-c1fa-452e-9116-deaeb697f6f7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:37.029596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:37.181085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e18a6157-c1fa-452e-9116-deaeb697f6f7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:37.266952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:37.301774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb9b2f3-fb90-43e9-a506-9e801bb65c60 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:37.302404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb9b2f3-fb90-43e9-a506-9e801bb65c60 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:37.302672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb9b2f3-fb90-43e9-a506-9e801bb65c60 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:37.302936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb9b2f3-fb90-43e9-a506-9e801bb65c60 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:37.304507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3bb9b2f3-fb90-43e9-a506-9e801bb65c60 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:37.399041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb9b2f3-fb90-43e9-a506-9e801bb65c60 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:37.400655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb9b2f3-fb90-43e9-a506-9e801bb65c60 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:37.418771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 743/1481] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:39.755406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ce36a8-170e-46d3-8ac7-5af995c412ca None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:39.756006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ce36a8-170e-46d3-8ac7-5af995c412ca None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:39.756006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59ce36a8-170e-46d3-8ac7-5af995c412ca None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:39.757326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 739/1482] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:07:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:39.764212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141e5148-eabb-4f58-8963-628dd19a55eb None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:39.764999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141e5148-eabb-4f58-8963-628dd19a55eb None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:39.764999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141e5148-eabb-4f58-8963-628dd19a55eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:39.840757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-141e5148-eabb-4f58-8963-628dd19a55eb None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:39.841981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-141e5148-eabb-4f58-8963-628dd19a55eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:39.890230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:39.942013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 744/1483] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:07:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:40.231668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ce36a8-170e-46d3-8ac7-5af995c412ca None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:40.233207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-59ce36a8-170e-46d3-8ac7-5af995c412ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:40.260533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:40.352067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59ce36a8-170e-46d3-8ac7-5af995c412ca None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:40.437486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b68b74d-1cba-4aa9-8a13-eb0032064333 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:40.438354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b68b74d-1cba-4aa9-8a13-eb0032064333 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:40.438714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b68b74d-1cba-4aa9-8a13-eb0032064333 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:40.439084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b68b74d-1cba-4aa9-8a13-eb0032064333 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:40.441779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4b68b74d-1cba-4aa9-8a13-eb0032064333 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:40.518898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:40.559507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b68b74d-1cba-4aa9-8a13-eb0032064333 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:40.560974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b68b74d-1cba-4aa9-8a13-eb0032064333 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:40.581287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 740/1484] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:07:42.837268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d573c-bf7e-4b86-842b-33070638fa21 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:42.838216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d573c-bf7e-4b86-842b-33070638fa21 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:42.838216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0d573c-bf7e-4b86-842b-33070638fa21 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:42.840348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 745/1485] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:07:42 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:42.849544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003804a3-5580-4b3b-b911-5ebcfaff8c88 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:42.850042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003804a3-5580-4b3b-b911-5ebcfaff8c88 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:42.850149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-003804a3-5580-4b3b-b911-5ebcfaff8c88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:42.921045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-003804a3-5580-4b3b-b911-5ebcfaff8c88 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:42.922159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-003804a3-5580-4b3b-b911-5ebcfaff8c88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:43.039019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 741/1486] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:07:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:43.326738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d573c-bf7e-4b86-842b-33070638fa21 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:43.328278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3b0d573c-bf7e-4b86-842b-33070638fa21 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:43.438057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0d573c-bf7e-4b86-842b-33070638fa21 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:43.513473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:43.514906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a41d9b-f78a-4482-bae4-9c00ce82b595 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:43.515517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a41d9b-f78a-4482-bae4-9c00ce82b595 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:43.515796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a41d9b-f78a-4482-bae4-9c00ce82b595 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:43.516067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a41d9b-f78a-4482-bae4-9c00ce82b595 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:43.518194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-57a41d9b-f78a-4482-bae4-9c00ce82b595 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:43.654492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a41d9b-f78a-4482-bae4-9c00ce82b595 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:43.655834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a41d9b-f78a-4482-bae4-9c00ce82b595 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:43.674727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 746/1487] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:46.051209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c54d58-2cd1-4ce2-9ece-f6b9679f1315 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:46.051807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c54d58-2cd1-4ce2-9ece-f6b9679f1315 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:46.051807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c54d58-2cd1-4ce2-9ece-f6b9679f1315 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:46.053937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 742/1488] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:07:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:46.060683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1806cfa-aac6-4a30-8c5f-c3609fd00c43 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:46.061126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1806cfa-aac6-4a30-8c5f-c3609fd00c43 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:46.061227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1806cfa-aac6-4a30-8c5f-c3609fd00c43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:46.142422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e1806cfa-aac6-4a30-8c5f-c3609fd00c43 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:46.144177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e1806cfa-aac6-4a30-8c5f-c3609fd00c43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:46.202588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:46.255961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 747/1489] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:07:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:07:46.544100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c54d58-2cd1-4ce2-9ece-f6b9679f1315 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:46.545718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e9c54d58-2cd1-4ce2-9ece-f6b9679f1315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:46.587073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:46.658707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c54d58-2cd1-4ce2-9ece-f6b9679f1315 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:46.737784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec64d33a-a86c-4f6f-8bcf-a7c15ad05f3d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:46.738413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec64d33a-a86c-4f6f-8bcf-a7c15ad05f3d service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:46.738688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec64d33a-a86c-4f6f-8bcf-a7c15ad05f3d service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:46.738948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec64d33a-a86c-4f6f-8bcf-a7c15ad05f3d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:46.740520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ec64d33a-a86c-4f6f-8bcf-a7c15ad05f3d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:46.826468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:46.857755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec64d33a-a86c-4f6f-8bcf-a7c15ad05f3d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:46.859171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec64d33a-a86c-4f6f-8bcf-a7c15ad05f3d service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:46.878496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 743/1490] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:49.211962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f545a1-efa2-4050-ac6e-a527197b8efa None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:49.212881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f545a1-efa2-4050-ac6e-a527197b8efa None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:49.212881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63f545a1-efa2-4050-ac6e-a527197b8efa None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:49.214957 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 748/1491] 10.222.0.61 () {54 vars in 927 bytes} [Thu Sep 29 02:07:49 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:07:49.220631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fc4fc8-6074-4536-adbd-7d15ca101e8b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:49.221115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fc4fc8-6074-4536-adbd-7d15ca101e8b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:49.221115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67fc4fc8-6074-4536-adbd-7d15ca101e8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:49.292960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-67fc4fc8-6074-4536-adbd-7d15ca101e8b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:07:49.294193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-67fc4fc8-6074-4536-adbd-7d15ca101e8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:49.399854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 744/1492] 10.222.0.61 () {58 vars in 1024 bytes} [Thu Sep 29 02:07:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:07:49.689003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f545a1-efa2-4050-ac6e-a527197b8efa None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:07:49.691404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-63f545a1-efa2-4050-ac6e-a527197b8efa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:49.805946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63f545a1-efa2-4050-ac6e-a527197b8efa None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:07:49.889300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d034457-7010-4693-8308-9fff52906edd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:07:49.889903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d034457-7010-4693-8308-9fff52906edd service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:07:49.890212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d034457-7010-4693-8308-9fff52906edd service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:07:49.890478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d034457-7010-4693-8308-9fff52906edd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:07:49.892060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1d034457-7010-4693-8308-9fff52906edd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:07:49.900439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:07:50.010670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d034457-7010-4693-8308-9fff52906edd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:07:50.012122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d034457-7010-4693-8308-9fff52906edd service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:07:50.031744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 749/1493] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:07:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:07:59.398270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=82 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:08:00.029236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=50 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:03.360902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:03.361661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:03.361661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:03.440075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:03.441282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:03.548665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 745/1494] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:09:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:03.555431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d140d5-82a5-44cc-a270-9ca48497f7ad None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:03.555915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d140d5-82a5-44cc-a270-9ca48497f7ad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:03.555915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6d140d5-82a5-44cc-a270-9ca48497f7ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:03.635143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b6d140d5-82a5-44cc-a270-9ca48497f7ad None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:03.636286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b6d140d5-82a5-44cc-a270-9ca48497f7ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:03.748048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 750/1495] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:09:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:09:03.753133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:03.754485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:03.766346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f09621b2-4c60-4767-9783-a36aaad0680a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:03.839433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:03.876716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:03.877333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:03.877607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:03.878151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:03.878662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:09:03.878969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:03.879992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:03.879992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:09:03.879992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:09:03.880878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:03.892825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 746/1496] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:09:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:09:03.902406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d140d5-82a5-44cc-a270-9ca48497f7ad None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:03.916656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6d140d5-82a5-44cc-a270-9ca48497f7ad None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:03.992697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:04.026827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:04.027430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:04.027703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:04.027989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:04.028599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:04.030391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:04.030391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 29 02:09:04.030391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: Sep 29 02:09:04.031591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:04.091908 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 751/1497] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:09:03 2022] POST /identity/v3/projects => generated 379 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:09:04.096406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:04.098029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:04.132533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d66dc61-2bec-487b-bb20-c8eb0d8729a0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:04.233804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:04.234377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:04.234631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:04.234896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:04.235532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:04.236556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:04.236556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:09:04.236556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:09:04.237842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:04.282890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 747/1498] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:09:04 2022] POST /identity/v3/users => generated 290 bytes in 187 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:09:04.287342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:04.288848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:04.363269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b867cb48-b101-4244-bdab-0c229e3e210d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:04.456199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:04.468538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:04.469159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:04.469433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:04.469694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:04.470360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:04.471353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:04.471353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 29 02:09:04.471353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: Sep 29 02:09:04.472459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:04.485184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 752/1499] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:09:04.489672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:04.491245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:04.502871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df390895-e553-43e2-bbaf-af965f385590 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:04.530166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:04.606459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90eb7ba4-0184-4897-8b3e-a2c4186e6483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:04.607066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eb7ba4-0184-4897-8b3e-a2c4186e6483 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:04.607307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eb7ba4-0184-4897-8b3e-a2c4186e6483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:04.607605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90eb7ba4-0184-4897-8b3e-a2c4186e6483 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:04.608239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eb7ba4-0184-4897-8b3e-a2c4186e6483 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:04.609256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-90eb7ba4-0184-4897-8b3e-a2c4186e6483 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:04.609256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:09:04.609256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:09:04.610166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90eb7ba4-0184-4897-8b3e-a2c4186e6483 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:04.623131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 748/1500] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:09:04 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:09:04.627579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:04.629209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:04.667168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739391c9-9d55-46ae-b04e-cc402499a467 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:04.775481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff97daa2-183a-4c06-93e5-9ab3c3c1fcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:04.776042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff97daa2-183a-4c06-93e5-9ab3c3c1fcd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:04.776312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff97daa2-183a-4c06-93e5-9ab3c3c1fcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:04.776602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff97daa2-183a-4c06-93e5-9ab3c3c1fcd3 admin admin] PATH_INFO: `/v3/projects/ca78ef8e5e4948ac80bdb236a27aa313/users/492d7f4e90fb4cc5956ca9376fe20a59/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:04.801010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff97daa2-183a-4c06-93e5-9ab3c3c1fcd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca78ef8e5e4948ac80bdb236a27aa313, user_id=492d7f4e90fb4cc5956ca9376fe20a59, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:04.802035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-ff97daa2-183a-4c06-93e5-9ab3c3c1fcd3 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:04.802035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 29 02:09:04.802035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: Sep 29 02:09:04.804002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff97daa2-183a-4c06-93e5-9ab3c3c1fcd3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:04.833727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 753/1501] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:09:04 2022] PUT /identity/v3/projects/ca78ef8e5e4948ac80bdb236a27aa313/users/492d7f4e90fb4cc5956ca9376fe20a59/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:09:04.846187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbbb311-a7b3-4e7a-a7cb-24796286c4b9 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:04.846466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbbb311-a7b3-4e7a-a7cb-24796286c4b9 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:04.846808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fbbb311-a7b3-4e7a-a7cb-24796286c4b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:04.904314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2fbbb311-a7b3-4e7a-a7cb-24796286c4b9 None None] MFA Rules not processed for user `492d7f4e90fb4cc5956ca9376fe20a59`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:04.905511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2fbbb311-a7b3-4e7a-a7cb-24796286c4b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:04.986953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 749/1502] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:09:04 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:05.000963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfe42c-ab55-46ec-a2ac-48e8ceda5516 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:05.001247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfe42c-ab55-46ec-a2ac-48e8ceda5516 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:05.001518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7abfe42c-ab55-46ec-a2ac-48e8ceda5516 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:05.003069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 754/1503] 10.222.0.61 () {54 vars in 963 bytes} [Thu Sep 29 02:09:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:05.007946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2856039-b8e7-462c-8d43-811ffc5c7c1c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:05.008226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2856039-b8e7-462c-8d43-811ffc5c7c1c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:05.008486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2856039-b8e7-462c-8d43-811ffc5c7c1c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:05.042725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:05.090922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f2856039-b8e7-462c-8d43-811ffc5c7c1c None None] MFA Rules not processed for user `36c4d127aac743baaeb86157dbbeb689`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:05.092124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f2856039-b8e7-462c-8d43-811ffc5c7c1c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:05.173092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 750/1504] 10.222.0.61 () {58 vars in 1060 bytes} [Thu Sep 29 02:09:05 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:05.184653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfe42c-ab55-46ec-a2ac-48e8ceda5516 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:05.186199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7abfe42c-ab55-46ec-a2ac-48e8ceda5516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:05.240795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:05.303728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7abfe42c-ab55-46ec-a2ac-48e8ceda5516 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:05.381611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:05.382239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:05.382508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:05.382773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:05.384362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:05.507043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:05.508485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:05.513162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:05.527294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 755/1505] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:05 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 343 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:05.573607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af674b3f-fba5-403b-ac59-a9ab3b90fe93 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:05.574115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af674b3f-fba5-403b-ac59-a9ab3b90fe93 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:05.574213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af674b3f-fba5-403b-ac59-a9ab3b90fe93 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:05.575478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 751/1506] 10.222.0.61 () {52 vars in 830 bytes} [Thu Sep 29 02:09:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:09:05.579392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:05.580614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:05.700342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e79822-65d8-4f18-9d72-5431e02840ab service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:05.806620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:05.807165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:05.807433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:05.807701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] PATH_INFO: `/v3/users/492d7f4e90fb4cc5956ca9376fe20a59` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:05.821781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] RBAC: Authorizing `identity:delete_user(user_id=492d7f4e90fb4cc5956ca9376fe20a59)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:05.822856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:05.822856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 29 02:09:05.822856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: Sep 29 02:09:05.824434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:05.836257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:05.898119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '492d7f4e90fb4cc5956ca9376fe20a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:05.913088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '492d7f4e90fb4cc5956ca9376fe20a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:05.913538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '492d7f4e90fb4cc5956ca9376fe20a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:05.914106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '492d7f4e90fb4cc5956ca9376fe20a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:05.925333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '492d7f4e90fb4cc5956ca9376fe20a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:05.960318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 756/1507] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:09:05 2022] DELETE /identity/v3/users/492d7f4e90fb4cc5956ca9376fe20a59 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:09:05.965036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:05.965319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:05.965591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:06.026175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:09:06.027385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:06.109404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:06.121869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 752/1508] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:09:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:09:06.133440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:06.134721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:06.237607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88512e2d-1b02-4d31-b06d-79ee67e59964 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:06.304110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:06.334030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88902e9b-8416-434b-b040-8691473e571f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:06.334660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88902e9b-8416-434b-b040-8691473e571f service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:06.334927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88902e9b-8416-434b-b040-8691473e571f service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:06.335194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88902e9b-8416-434b-b040-8691473e571f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:06.336780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-88902e9b-8416-434b-b040-8691473e571f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:06.459409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88902e9b-8416-434b-b040-8691473e571f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:06.460882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88902e9b-8416-434b-b040-8691473e571f service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:06.481666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 757/1509] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:09:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:09:06.852126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:09:06.853365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:09:06.865314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-803a1eee-8ab3-4ead-a0ef-ee537a9b13fc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:09:06.966848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:09:06.967490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:09:06.967761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:09:06.968026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] PATH_INFO: `/v3/projects/ca78ef8e5e4948ac80bdb236a27aa313` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:09:06.979418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca78ef8e5e4948ac80bdb236a27aa313)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:09:06.980468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:09:06.980468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:09:06.980468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:09:06.981673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:09:07.053452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca78ef8e5e4948ac80bdb236a27aa313', 'request_id': 'req-7f56dc5d-288b-4a83-bd0d-67f81492c00b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:07.064694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca78ef8e5e4948ac80bdb236a27aa313', 'request_id': 'req-7f56dc5d-288b-4a83-bd0d-67f81492c00b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:07.079111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca78ef8e5e4948ac80bdb236a27aa313', 'request_id': 'req-7f56dc5d-288b-4a83-bd0d-67f81492c00b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:07.101169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] The token cache is being invalidate because project ca78ef8e5e4948ac80bdb236a27aa313 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:09:07.101452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:07.101872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7f56dc5d-288b-4a83-bd0d-67f81492c00b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:09:07.103306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 753/1510] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:09:06 2022] DELETE /identity/v3/projects/ca78ef8e5e4948ac80bdb236a27aa313 => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:09:16.478925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=62 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:09:17.099683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:10:36.031138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ef4ff8-730a-4779-9fdd-4d492f58533d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:36.031787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ef4ff8-730a-4779-9fdd-4d492f58533d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:36.031787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4ef4ff8-730a-4779-9fdd-4d492f58533d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:36.033199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 758/1511] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:10:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:10:36.038861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147be8a6-d1b4-4a09-85f1-c53132881c48 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:36.039320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147be8a6-d1b4-4a09-85f1-c53132881c48 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:36.039430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147be8a6-d1b4-4a09-85f1-c53132881c48 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:36.128975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-147be8a6-d1b4-4a09-85f1-c53132881c48 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:36.130208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-147be8a6-d1b4-4a09-85f1-c53132881c48 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:36.235286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 754/1512] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:10:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:10:36.392046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ef4ff8-730a-4779-9fdd-4d492f58533d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:10:36.395221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c4ef4ff8-730a-4779-9fdd-4d492f58533d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:36.524733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4ef4ff8-730a-4779-9fdd-4d492f58533d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:10:36.531578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef67e149-fcfe-4320-b916-0f457852869b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:10:36.532163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef67e149-fcfe-4320-b916-0f457852869b service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:36.532434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef67e149-fcfe-4320-b916-0f457852869b service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:36.532673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef67e149-fcfe-4320-b916-0f457852869b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:36.541583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef67e149-fcfe-4320-b916-0f457852869b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:10:36.543209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef67e149-fcfe-4320-b916-0f457852869b service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:10:36.552909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 759/1513] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:10:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:10:37.629710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5647a98-03b3-433a-bb4d-13e54bea9de6 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:37.630390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5647a98-03b3-433a-bb4d-13e54bea9de6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:37.630390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5647a98-03b3-433a-bb4d-13e54bea9de6 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:37.631636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 755/1514] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:10:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:10:37.639043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc88638-0bad-418b-909a-5541094533ac None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:37.639574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc88638-0bad-418b-909a-5541094533ac None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:37.639574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc88638-0bad-418b-909a-5541094533ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:37.661059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0dc88638-0bad-418b-909a-5541094533ac None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:37.662330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0dc88638-0bad-418b-909a-5541094533ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:37.669266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18174844-25dc-4d26-9093-faef74b703b5 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:37.669569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18174844-25dc-4d26-9093-faef74b703b5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:37.669839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18174844-25dc-4d26-9093-faef74b703b5 None None] PATH_INFO: `/v3` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:37.671131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 756/1515] 10.222.0.55 () {54 vars in 925 bytes} [Thu Sep 29 02:10:37 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:10:37.672495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 760/1516] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:10:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:10:37.677793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb0385-70b0-4b39-aa16-b00493b85a7f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:37.678210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb0385-70b0-4b39-aa16-b00493b85a7f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:37.679259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb0385-70b0-4b39-aa16-b00493b85a7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:37.740563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-e0cb0385-70b0-4b39-aa16-b00493b85a7f None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:37.741750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e0cb0385-70b0-4b39-aa16-b00493b85a7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:37.772021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:10:37.815642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 757/1517] 10.222.0.55 () {58 vars in 1015 bytes} [Thu Sep 29 02:10:37 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:10:37.824718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc88638-0bad-418b-909a-5541094533ac None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:10:37.837460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc88638-0bad-418b-909a-5541094533ac None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:10:37.842812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0ee7851-f3f8-44c5-a927-5a819aedf673 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:10:37.843402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ee7851-f3f8-44c5-a927-5a819aedf673 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:37.843671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ee7851-f3f8-44c5-a927-5a819aedf673 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:37.843936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0ee7851-f3f8-44c5-a927-5a819aedf673 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:37.851960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ee7851-f3f8-44c5-a927-5a819aedf673 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:10:37.853631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0ee7851-f3f8-44c5-a927-5a819aedf673 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:10:37.862875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 761/1518] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:10:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:10:46.875253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac2e865-2866-4e4b-acc4-2853ee839afc None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:46.876056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac2e865-2866-4e4b-acc4-2853ee839afc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:46.876056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac2e865-2866-4e4b-acc4-2853ee839afc None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:46.878163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 758/1519] 10.222.0.61 () {54 vars in 961 bytes} [Thu Sep 29 02:10:46 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:10:46.887067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6163ec91-bba8-4f88-b46f-f84dffa57f29 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:46.887533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6163ec91-bba8-4f88-b46f-f84dffa57f29 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:46.887687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6163ec91-bba8-4f88-b46f-f84dffa57f29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:46.912175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-6163ec91-bba8-4f88-b46f-f84dffa57f29 None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:46.913390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6163ec91-bba8-4f88-b46f-f84dffa57f29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:46.923565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 762/1520] 10.222.0.61 () {58 vars in 1058 bytes} [Thu Sep 29 02:10:46 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:10:46.931864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac2e865-2866-4e4b-acc4-2853ee839afc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:10:46.947187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cac2e865-2866-4e4b-acc4-2853ee839afc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:10:46.954178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3119d4ae-7293-4bff-9eac-72507e0b3110 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:10:46.954795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3119d4ae-7293-4bff-9eac-72507e0b3110 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:46.955394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3119d4ae-7293-4bff-9eac-72507e0b3110 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:46.955666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3119d4ae-7293-4bff-9eac-72507e0b3110 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:46.964829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3119d4ae-7293-4bff-9eac-72507e0b3110 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:10:46.966465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3119d4ae-7293-4bff-9eac-72507e0b3110 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:10:46.976683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 759/1521] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:10:46 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:10:49.354367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cf39e2-129d-4035-b7c6-d1fbbf644819 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:49.354982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cf39e2-129d-4035-b7c6-d1fbbf644819 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:49.354982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4cf39e2-129d-4035-b7c6-d1fbbf644819 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:49.356272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 763/1522] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:10:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:10:49.362691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f673d2-f6e4-4aee-a950-4e3124bcd52b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:49.363155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f673d2-f6e4-4aee-a950-4e3124bcd52b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:49.363251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f673d2-f6e4-4aee-a950-4e3124bcd52b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:49.385174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-30f673d2-f6e4-4aee-a950-4e3124bcd52b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:49.386398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-30f673d2-f6e4-4aee-a950-4e3124bcd52b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:49.394713 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 760/1523] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:10:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:10:49.552562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cf39e2-129d-4035-b7c6-d1fbbf644819 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:10:49.568391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4cf39e2-129d-4035-b7c6-d1fbbf644819 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:10:49.574291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d4cc495-873c-4c1b-9c51-8df64d2d485b service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:10:49.574890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4cc495-873c-4c1b-9c51-8df64d2d485b service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:49.575166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4cc495-873c-4c1b-9c51-8df64d2d485b service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:49.575440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d4cc495-873c-4c1b-9c51-8df64d2d485b service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:49.584038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4cc495-873c-4c1b-9c51-8df64d2d485b service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:10:49.585640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d4cc495-873c-4c1b-9c51-8df64d2d485b service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:10:49.595451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 764/1524] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:10:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:10:50.660325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a0669-43d2-4789-9d50-4c715a9c270f None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:50.661184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a0669-43d2-4789-9d50-4c715a9c270f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:50.661184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959a0669-43d2-4789-9d50-4c715a9c270f None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:50.663409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 761/1525] 127.0.0.1 () {54 vars in 901 bytes} [Thu Sep 29 02:10:50 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:10:50.669966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d632264-6f90-41aa-a133-6c46a372464e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:50.670497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d632264-6f90-41aa-a133-6c46a372464e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:50.670497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d632264-6f90-41aa-a133-6c46a372464e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:50.691622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0d632264-6f90-41aa-a133-6c46a372464e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:50.692865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0d632264-6f90-41aa-a133-6c46a372464e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:50.701886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 765/1526] 127.0.0.1 () {58 vars in 998 bytes} [Thu Sep 29 02:10:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:10:50.857526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959a0669-43d2-4789-9d50-4c715a9c270f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:10:50.873254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959a0669-43d2-4789-9d50-4c715a9c270f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:10:50.881304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62135c37-d314-4bb8-9791-68fc2c2627a7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:10:50.882345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62135c37-d314-4bb8-9791-68fc2c2627a7 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:50.882345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62135c37-d314-4bb8-9791-68fc2c2627a7 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:50.882997 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62135c37-d314-4bb8-9791-68fc2c2627a7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:50.891854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62135c37-d314-4bb8-9791-68fc2c2627a7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:10:50.893446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62135c37-d314-4bb8-9791-68fc2c2627a7 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:10:50.903273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 762/1527] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:10:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:10:55.107954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c649e75-b561-44e6-b954-df7d1de1408e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:55.108569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c649e75-b561-44e6-b954-df7d1de1408e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:55.108569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c649e75-b561-44e6-b954-df7d1de1408e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:55.109860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 766/1528] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:10:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:10:55.116344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea34b99d-65ae-4b7a-b5e3-91e7b506cd33 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:55.116808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea34b99d-65ae-4b7a-b5e3-91e7b506cd33 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:55.116928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea34b99d-65ae-4b7a-b5e3-91e7b506cd33 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:55.139606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ea34b99d-65ae-4b7a-b5e3-91e7b506cd33 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:55.140829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ea34b99d-65ae-4b7a-b5e3-91e7b506cd33 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:55.150244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 763/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:10:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:10:55.441402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c649e75-b561-44e6-b954-df7d1de1408e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:10:55.444049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2c649e75-b561-44e6-b954-df7d1de1408e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:55.511644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:10:55.527520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c649e75-b561-44e6-b954-df7d1de1408e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:10:55.531598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d20f119-6fdf-4414-a964-5c62da97973c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:10:55.532206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d20f119-6fdf-4414-a964-5c62da97973c service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:55.532479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d20f119-6fdf-4414-a964-5c62da97973c service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:55.532746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d20f119-6fdf-4414-a964-5c62da97973c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:55.542482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d20f119-6fdf-4414-a964-5c62da97973c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:10:55.543941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d20f119-6fdf-4414-a964-5c62da97973c service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:10:55.554208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 767/1530] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:10:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:10:57.667160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6c9a2-4dc3-4627-9283-1ebda7a28af5 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:57.667722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6c9a2-4dc3-4627-9283-1ebda7a28af5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:57.667722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38f6c9a2-4dc3-4627-9283-1ebda7a28af5 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:57.669016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 764/1531] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:10:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:10:57.674380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45fdc4a-02d8-4a41-b283-6b6721d43d6c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:57.674833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45fdc4a-02d8-4a41-b283-6b6721d43d6c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:57.674966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45fdc4a-02d8-4a41-b283-6b6721d43d6c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:57.695600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a45fdc4a-02d8-4a41-b283-6b6721d43d6c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:10:57.697194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a45fdc4a-02d8-4a41-b283-6b6721d43d6c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:10:57.705447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 768/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:10:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:10:57.992643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f6c9a2-4dc3-4627-9283-1ebda7a28af5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:10:58.005604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38f6c9a2-4dc3-4627-9283-1ebda7a28af5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:10:58.009801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b307a049-f62f-4c85-b4bb-9e252a392cb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:10:58.010412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b307a049-f62f-4c85-b4bb-9e252a392cb8 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:10:58.010681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b307a049-f62f-4c85-b4bb-9e252a392cb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:10:58.010944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b307a049-f62f-4c85-b4bb-9e252a392cb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:10:58.020823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b307a049-f62f-4c85-b4bb-9e252a392cb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:10:58.022281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b307a049-f62f-4c85-b4bb-9e252a392cb8 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:10:58.031874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 765/1533] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:10:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:11:01.815518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f4b2c-4aef-4ef9-9d4a-7e5709c1941d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:01.816303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f4b2c-4aef-4ef9-9d4a-7e5709c1941d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:01.816303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525f4b2c-4aef-4ef9-9d4a-7e5709c1941d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:01.818369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 769/1534] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:11:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:01.823880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2dd9b3-c4f1-4672-889b-497a3905c96a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:01.824368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2dd9b3-c4f1-4672-889b-497a3905c96a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:01.824477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2dd9b3-c4f1-4672-889b-497a3905c96a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:01.845624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2d2dd9b3-c4f1-4672-889b-497a3905c96a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:01.846841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2d2dd9b3-c4f1-4672-889b-497a3905c96a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:01.856756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 766/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:11:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:11:02.166789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525f4b2c-4aef-4ef9-9d4a-7e5709c1941d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:02.168609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-525f4b2c-4aef-4ef9-9d4a-7e5709c1941d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:02.254823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525f4b2c-4aef-4ef9-9d4a-7e5709c1941d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:02.258888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9969ea0f-f822-42d6-ac45-9596b3b3d7cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:02.259735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9969ea0f-f822-42d6-ac45-9596b3b3d7cc service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:02.259883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9969ea0f-f822-42d6-ac45-9596b3b3d7cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:02.260142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9969ea0f-f822-42d6-ac45-9596b3b3d7cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:02.269386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9969ea0f-f822-42d6-ac45-9596b3b3d7cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:02.270848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9969ea0f-f822-42d6-ac45-9596b3b3d7cc service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:02.280494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 770/1536] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:11:04.421133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43efeceb-6320-45b5-92e3-a38632870132 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:04.421726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43efeceb-6320-45b5-92e3-a38632870132 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:04.421726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43efeceb-6320-45b5-92e3-a38632870132 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:04.423845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 767/1537] 127.0.0.1 () {54 vars in 917 bytes} [Thu Sep 29 02:11:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:11:04.431823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c86b87-0261-4642-a66b-fa74e48af9fe None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:04.432328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c86b87-0261-4642-a66b-fa74e48af9fe None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:04.432328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c86b87-0261-4642-a66b-fa74e48af9fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:04.454376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-31c86b87-0261-4642-a66b-fa74e48af9fe None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:11:04.456071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-31c86b87-0261-4642-a66b-fa74e48af9fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:04.465830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 771/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Thu Sep 29 02:11:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:11:04.772375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43efeceb-6320-45b5-92e3-a38632870132 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:11:04.774842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-43efeceb-6320-45b5-92e3-a38632870132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:11:04.794295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43efeceb-6320-45b5-92e3-a38632870132 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:11:04.799693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c245c37-a775-493c-8c58-15b827f50a68 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:11:04.800286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c245c37-a775-493c-8c58-15b827f50a68 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:11:04.800564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c245c37-a775-493c-8c58-15b827f50a68 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:11:04.800824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c245c37-a775-493c-8c58-15b827f50a68 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:11:04.812629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c245c37-a775-493c-8c58-15b827f50a68 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:11:04.814517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c245c37-a775-493c-8c58-15b827f50a68 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:11:04.826573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 768/1539] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:11:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:11:14.448783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=50 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:11:14.819991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=50 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:28.989688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15d95eb-db64-4fed-87a0-5a813772dd19 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:28.990418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15d95eb-db64-4fed-87a0-5a813772dd19 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:28.990418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d15d95eb-db64-4fed-87a0-5a813772dd19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.029847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d15d95eb-db64-4fed-87a0-5a813772dd19 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.035570 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d15d95eb-db64-4fed-87a0-5a813772dd19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.044384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 772/1540] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (3 switches on core 0) Sep 29 02:12:29.053548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb534153-2f25-4793-9fd0-7ee4298c9f35 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.054169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb534153-2f25-4793-9fd0-7ee4298c9f35 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.054169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb534153-2f25-4793-9fd0-7ee4298c9f35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.083927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-bb534153-2f25-4793-9fd0-7ee4298c9f35 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.085160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bb534153-2f25-4793-9fd0-7ee4298c9f35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.101126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 769/1541] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.105693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15d95eb-db64-4fed-87a0-5a813772dd19 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.119485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d15d95eb-db64-4fed-87a0-5a813772dd19 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.125684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.126319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.126589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.126854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.127499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:29.127799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.128815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:12:29.128815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 29 02:12:29.128815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: Sep 29 02:12:29.129698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.146847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 773/1542] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:29.155223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb534153-2f25-4793-9fd0-7ee4298c9f35 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.169789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb534153-2f25-4793-9fd0-7ee4298c9f35 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.182563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f592b2d4-067a-49d8-b4bc-9ba7019bde78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.183212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f592b2d4-067a-49d8-b4bc-9ba7019bde78 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.183488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f592b2d4-067a-49d8-b4bc-9ba7019bde78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.183754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f592b2d4-067a-49d8-b4bc-9ba7019bde78 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.184415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f592b2d4-067a-49d8-b4bc-9ba7019bde78 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.185491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-f592b2d4-067a-49d8-b4bc-9ba7019bde78 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:12:29.185491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:12:29.185491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:12:29.186755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f592b2d4-067a-49d8-b4bc-9ba7019bde78 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.255846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 770/1543] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/projects => generated 375 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:29.260573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.274660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5aff96d-6c5a-445f-a47c-c12a86a67fc7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.331624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211d12ce-8f07-489b-8f4d-6b86d30063a7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.332003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211d12ce-8f07-489b-8f4d-6b86d30063a7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.332179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-211d12ce-8f07-489b-8f4d-6b86d30063a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.353134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.354282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.354560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.354826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.355484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.356554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:12:29.356554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 29 02:12:29.356554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: Sep 29 02:12:29.357771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.366537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-211d12ce-8f07-489b-8f4d-6b86d30063a7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.367923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-211d12ce-8f07-489b-8f4d-6b86d30063a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.386784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 771/1544] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 56 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:29.392775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86085eb9-92d3-43f9-a341-08471531de18 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.393085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86085eb9-92d3-43f9-a341-08471531de18 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.393405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86085eb9-92d3-43f9-a341-08471531de18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.399160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 774/1545] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/users => generated 288 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:29.403193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.418646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d41c4fe-f6fc-4d84-abb9-8e6e755ef19e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.426977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-86085eb9-92d3-43f9-a341-08471531de18 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.428538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-86085eb9-92d3-43f9-a341-08471531de18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.431135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87a2bbdc-bed4-4c71-b339-1e3b8ead2c28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.431721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a2bbdc-bed4-4c71-b339-1e3b8ead2c28 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.431997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a2bbdc-bed4-4c71-b339-1e3b8ead2c28 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.432268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87a2bbdc-bed4-4c71-b339-1e3b8ead2c28 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.432918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a2bbdc-bed4-4c71-b339-1e3b8ead2c28 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.434622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87a2bbdc-bed4-4c71-b339-1e3b8ead2c28 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.444566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 772/1546] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:29.446088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e48e6-dca7-4769-adf3-35bec1ca09c5 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.446475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e48e6-dca7-4769-adf3-35bec1ca09c5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.446752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-986e48e6-dca7-4769-adf3-35bec1ca09c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.453273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 775/1547] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:12:29.454669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f776cb-6cca-473c-b194-c2fc36fc3557 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.454952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f776cb-6cca-473c-b194-c2fc36fc3557 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.455222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f776cb-6cca-473c-b194-c2fc36fc3557 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.480306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-986e48e6-dca7-4769-adf3-35bec1ca09c5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.481515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-986e48e6-dca7-4769-adf3-35bec1ca09c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.490353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 773/1548] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.494642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986e48e6-dca7-4769-adf3-35bec1ca09c5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.500340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-87f776cb-6cca-473c-b194-c2fc36fc3557 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.501548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-87f776cb-6cca-473c-b194-c2fc36fc3557 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.505948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-986e48e6-dca7-4769-adf3-35bec1ca09c5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.511124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 776/1549] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.512243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eaddac-29b8-404a-be98-9594cbd8054d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.512391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c56f6-4f38-463b-b733-be828d4ad1d3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.513030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c56f6-4f38-463b-b733-be828d4ad1d3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.513310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e07c56f6-4f38-463b-b733-be828d4ad1d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.513592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaddac-29b8-404a-be98-9594cbd8054d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.514180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaddac-29b8-404a-be98-9594cbd8054d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.514705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eaddac-29b8-404a-be98-9594cbd8054d admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.515537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eaddac-29b8-404a-be98-9594cbd8054d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.518145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eaddac-29b8-404a-be98-9594cbd8054d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.530704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 774/1550] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:29.532365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221410f-b841-4d4e-8b0e-7a5ced03e13d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.532884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221410f-b841-4d4e-8b0e-7a5ced03e13d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.533732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a221410f-b841-4d4e-8b0e-7a5ced03e13d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.546672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e07c56f6-4f38-463b-b733-be828d4ad1d3 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.548558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e07c56f6-4f38-463b-b733-be828d4ad1d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.556405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a221410f-b841-4d4e-8b0e-7a5ced03e13d None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.557647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a221410f-b841-4d4e-8b0e-7a5ced03e13d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.559261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 777/1551] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.561419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c079c0-ce8c-42b3-a503-d465bb9bb185 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.561969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c079c0-ce8c-42b3-a503-d465bb9bb185 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.562474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c079c0-ce8c-42b3-a503-d465bb9bb185 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.566403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 775/1552] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.567510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a221410f-b841-4d4e-8b0e-7a5ced03e13d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.578031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a221410f-b841-4d4e-8b0e-7a5ced03e13d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.582760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.583361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.583634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.583901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.584539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:29.584832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.585151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-78c079c0-ce8c-42b3-a503-d465bb9bb185 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.586460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-78c079c0-ce8c-42b3-a503-d465bb9bb185 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.586986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.595041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 778/1553] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.595621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c079c0-ce8c-42b3-a503-d465bb9bb185 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.602819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 776/1554] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:29.604227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.605448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c079c0-ce8c-42b3-a503-d465bb9bb185 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.610043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88f584b-70b8-47a4-9802-02bfd86442a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.610642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f584b-70b8-47a4-9802-02bfd86442a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.610912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f584b-70b8-47a4-9802-02bfd86442a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.611178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88f584b-70b8-47a4-9802-02bfd86442a0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.611815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88f584b-70b8-47a4-9802-02bfd86442a0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:29.612112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88f584b-70b8-47a4-9802-02bfd86442a0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.613704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88f584b-70b8-47a4-9802-02bfd86442a0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.614287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c04ca0c-477d-4382-a0bd-432274ebf81a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.619623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8baa15c-962c-49b8-8100-1539ea8d9433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.620275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8baa15c-962c-49b8-8100-1539ea8d9433 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.620551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8baa15c-962c-49b8-8100-1539ea8d9433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.620866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8baa15c-962c-49b8-8100-1539ea8d9433 admin admin] PATH_INFO: `/v3/projects/45259b9967b9410ca0fb09df7cea9f15/users/3b624208bb94427791219ffcca0efc41/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.625305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 779/1555] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:29.627509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcc1b10-4042-4598-96b4-1a5b0c06bfdd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.627788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcc1b10-4042-4598-96b4-1a5b0c06bfdd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.628058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bcc1b10-4042-4598-96b4-1a5b0c06bfdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.639111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8baa15c-962c-49b8-8100-1539ea8d9433 admin admin] RBAC: Authorizing `identity:create_grant(project_id=45259b9967b9410ca0fb09df7cea9f15, user_id=3b624208bb94427791219ffcca0efc41, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.640163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-e8baa15c-962c-49b8-8100-1539ea8d9433 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:12:29.640163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:12:29.640163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:12:29.642111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8baa15c-962c-49b8-8100-1539ea8d9433 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.647167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3bcc1b10-4042-4598-96b4-1a5b0c06bfdd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.648610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3bcc1b10-4042-4598-96b4-1a5b0c06bfdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.657200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 780/1556] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.658433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67820a52-af58-4c81-83a2-7b4894bcccb9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.658707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67820a52-af58-4c81-83a2-7b4894bcccb9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.658975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67820a52-af58-4c81-83a2-7b4894bcccb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.663101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 777/1557] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:29 2022] PUT /identity/v3/projects/45259b9967b9410ca0fb09df7cea9f15/users/3b624208bb94427791219ffcca0efc41/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:29.663101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a327d5-366b-4f53-abab-461fe5e80844 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.663101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a327d5-366b-4f53-abab-461fe5e80844 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.663101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a327d5-366b-4f53-abab-461fe5e80844 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.681043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-67820a52-af58-4c81-83a2-7b4894bcccb9 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.681266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-35a327d5-366b-4f53-abab-461fe5e80844 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.682284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-67820a52-af58-4c81-83a2-7b4894bcccb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.682392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-35a327d5-366b-4f53-abab-461fe5e80844 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.743245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 781/1558] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.744258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 778/1559] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.744757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf3d02e-c13c-4646-927b-0a8eff7f3942 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.745036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf3d02e-c13c-4646-927b-0a8eff7f3942 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.745303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcf3d02e-c13c-4646-927b-0a8eff7f3942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.745604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89882dbf-8fb2-434f-8b82-c778c7d7a541 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.745928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89882dbf-8fb2-434f-8b82-c778c7d7a541 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.746231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89882dbf-8fb2-434f-8b82-c778c7d7a541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.765887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-89882dbf-8fb2-434f-8b82-c778c7d7a541 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.766127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fcf3d02e-c13c-4646-927b-0a8eff7f3942 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.767020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fcf3d02e-c13c-4646-927b-0a8eff7f3942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.767159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-89882dbf-8fb2-434f-8b82-c778c7d7a541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.775288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 779/1560] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.776196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89882dbf-8fb2-434f-8b82-c778c7d7a541 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.779157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 782/1561] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.779959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf3d02e-c13c-4646-927b-0a8eff7f3942 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.787875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89882dbf-8fb2-434f-8b82-c778c7d7a541 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.789612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcf3d02e-c13c-4646-927b-0a8eff7f3942 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.792778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f4fa78-0687-4e71-a4ad-beca828887ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.793369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f4fa78-0687-4e71-a4ad-beca828887ee admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.793643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f4fa78-0687-4e71-a4ad-beca828887ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.793914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f4fa78-0687-4e71-a4ad-beca828887ee admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.794314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42dc5957-8975-4984-955f-129b454a20ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.794670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f4fa78-0687-4e71-a4ad-beca828887ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.794901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dc5957-8975-4984-955f-129b454a20ae admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.795159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dc5957-8975-4984-955f-129b454a20ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.795428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42dc5957-8975-4984-955f-129b454a20ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.796052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dc5957-8975-4984-955f-129b454a20ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.796610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84f4fa78-0687-4e71-a4ad-beca828887ee admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.797943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42dc5957-8975-4984-955f-129b454a20ae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.810689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:29.830693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 780/1562] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/projects => generated 371 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:29.830693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a355b76-0b31-4fc7-894a-988699ea0ee7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.830693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a355b76-0b31-4fc7-894a-988699ea0ee7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.830693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a355b76-0b31-4fc7-894a-988699ea0ee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.833231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 783/1563] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/projects => generated 377 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:29.833231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319bd726-526d-4547-87eb-0d505db07eb6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.833231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319bd726-526d-4547-87eb-0d505db07eb6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.833231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319bd726-526d-4547-87eb-0d505db07eb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.848581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1a355b76-0b31-4fc7-894a-988699ea0ee7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.849770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1a355b76-0b31-4fc7-894a-988699ea0ee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.855219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-319bd726-526d-4547-87eb-0d505db07eb6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.856401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-319bd726-526d-4547-87eb-0d505db07eb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.862985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:29.911390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 781/1564] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.913122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef1f5f-7ec6-4bcd-801f-e60d81986cfe None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.913617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef1f5f-7ec6-4bcd-801f-e60d81986cfe None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.914136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceef1f5f-7ec6-4bcd-801f-e60d81986cfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.933409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 784/1565] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.934877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319bd726-526d-4547-87eb-0d505db07eb6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.937081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ceef1f5f-7ec6-4bcd-801f-e60d81986cfe None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:29.938301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ceef1f5f-7ec6-4bcd-801f-e60d81986cfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:29.944681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-319bd726-526d-4547-87eb-0d505db07eb6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.946143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 782/1566] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:29.949878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceef1f5f-7ec6-4bcd-801f-e60d81986cfe None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.953537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.954177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.954453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.954722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.955367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:29.955666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.956614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceef1f5f-7ec6-4bcd-801f-e60d81986cfe None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.957530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.961062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.961673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.961946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.962659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.963294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:29.963592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.965183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.978168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 785/1567] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:29.978168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.978476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 783/1568] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:29.978476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:29.988261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c846bc3-59af-4a2b-9fa4-b366d0890cbc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.988513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5542da16-face-48cb-898e-e53378df8052 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:29.993144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.993672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:29.993815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.994080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.994205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:29.994357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.994462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:29.994794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:29.995196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:29.995414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.995517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:29.997141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:29.997341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.010720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 786/1569] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:30.011938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.024391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1994383-e4c4-4a50-8e07-2c9ad890d6e8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.029138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.029731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.030029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.030220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 784/1570] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:29 2022] POST /identity/v3/users => generated 286 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.030641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.031176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.031341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.033349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.057510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 787/1571] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/users => generated 289 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.058383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.068382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a2b361-4768-40ba-bc02-9ada5ce96830 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.073123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.073716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.073990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.074291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.074932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:30.075234 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.076827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.086765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c7f68f-fc30-439e-90b3-9b9e6d48fab3 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.089716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 788/1572] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:30.090555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.090765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.091396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.091665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.091953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.102974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.104423 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f654bc6b-3eef-4f82-be5d-0292424cc1b4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.104548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.109037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.109624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.109926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.110221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.110858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:30.111158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.112740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.114273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 785/1573] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:30.115037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.124672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd1572-ea5e-4c03-9b0a-b8ba86709789 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.124824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 789/1574] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:30.125607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.135533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d238bd9-f6db-4ed7-a0ed-000ad1c9ff7c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.138466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.139086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.139360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.139631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.140258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.140374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.140950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.141258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.141554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.142249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.142364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.144346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.173597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 786/1575] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/projects => generated 393 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.177616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.181970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 790/1576] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.181970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.252558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db1f00f3-3634-47d0-96e9-3c6a5085650f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.252711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c9c3e7-8000-4caf-a21f-a13393c8e962 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.257862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.258123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.258554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.258842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.259150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.259328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.259736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.259840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.260159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.260874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.261341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.262922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:30.263231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.274274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:30.279061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 791/1577] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.279061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.289919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26e42d64-50ec-4610-924c-f8dffba7f6f0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.295823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.296569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.296917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.297219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.302457 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.302602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 787/1578] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/projects => generated 389 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.302733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.304241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.314707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 792/1579] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.316045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.327587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d3d05db-8902-43ba-9257-737b694d081a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.377669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0df7afcc-5e89-4502-bc44-d3ca936f312a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.383402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.384194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.384473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.384785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.385415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.387550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.393531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.394340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.394706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.395125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.396063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.398079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.423647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 788/1580] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/projects => generated 385 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.424582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.429720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 793/1581] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/projects => generated 357 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.431476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.511749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9ada69-e6c1-40d4-b106-b983971b8cd5 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.515955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff21e1c3-cbd1-4fa7-b059-cfcd84eb21a3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.517046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.517663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.517939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.518372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.519011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.521347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.532452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.533555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.534179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.534823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.535997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.539671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.557802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 789/1582] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/users => generated 297 bytes in 134 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.559389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.574905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b52625f3-a14f-42b7-9c09-6e0b4693f621 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.576447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 794/1583] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/users => generated 293 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.577560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.582066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.582750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.583105 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.583391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.584021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.585691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.593963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebafc021-6877-41c8-94de-ebaafe5fae6d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.597671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 790/1584] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.599817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.611752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2968bd-6cd6-4592-b72f-769bece6a4ed admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.612892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.613824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.614140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.614414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.615066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.616743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.617129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.617520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.617878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.618434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.619151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.620868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.632182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 791/1585] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.633445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.645862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f1b413-3c8f-4847-b8ab-66d031f22747 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.646482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 795/1586] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/users => generated 295 bytes in 69 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.647397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.652524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.653227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.653583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.653860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.654851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.656652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.658672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5eb7fc-3b20-4c5c-b91e-e85df1565953 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.663398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.664134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.664546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.664904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.665805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.667852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.682433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 792/1587] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/users => generated 293 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.683629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.694382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809d94e1-0f9a-48db-a3d0-4f3dbf847e23 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.702664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 796/1588] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:30 2022] POST /identity/v3/users => generated 279 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:30.702851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.702851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.703124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.703222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.703222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.703222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.703482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.718481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 793/1589] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.718706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5f2221-4a3f-4cb4-8b7a-48920096a41f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.718823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.723853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.724287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.724626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.724962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.725662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.727530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.729232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1333ddb-c079-4112-9b45-f8259e4b16c6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.733844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.734541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.734816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.735089 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] PATH_INFO: `/v3/projects/3aa3d6d0a4b14baab0999c7c69f44ea4/users/1202a5b40a1d4c52bf5a8fe965f5f35e/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.738654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 797/1590] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.742618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.748496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:30.753304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3aa3d6d0a4b14baab0999c7c69f44ea4, user_id=1202a5b40a1d4c52bf5a8fe965f5f35e, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.753492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f67371e-e77c-4ff1-840f-0364deff945d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.756941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.759373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.760102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.763057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.763057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] PATH_INFO: `/v3/projects/e72ef47313ef4a8582a9e060d5597fc0/users/480c3f60b5e14b5f93e5dc921694aee4/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.779494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 794/1591] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:30 2022] PUT /identity/v3/projects/3aa3d6d0a4b14baab0999c7c69f44ea4/users/1202a5b40a1d4c52bf5a8fe965f5f35e/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:30.781205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:INSERT=10 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:30.783354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.795614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e72ef47313ef4a8582a9e060d5597fc0, user_id=480c3f60b5e14b5f93e5dc921694aee4, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.799103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.818982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 798/1592] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:30 2022] PUT /identity/v3/projects/e72ef47313ef4a8582a9e060d5597fc0/users/480c3f60b5e14b5f93e5dc921694aee4/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:30.823840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.882739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85320a7f-8753-49e1-8154-4017adba45c0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.906585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-516c5eb8-7df3-4f4a-824d-43629481d729 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.912187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.913310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.913935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.914445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.915206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.917135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.931838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 799/1593] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.936002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.950278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-706a56fe-e504-4108-b863-f17c2f28af75 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.954689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.955379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.955509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.955823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.956162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.956436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.956776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.957004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.957129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.957971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.958926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.962065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.972344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 795/1594] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.973342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.977773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 800/1595] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:30.978770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:30.984920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9683441-843c-493c-a8e6-01f88b86a9bb admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:30.990106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb4b0f2-76ed-465c-8a1e-48c0e96d998e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:30.990834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4b0f2-76ed-465c-8a1e-48c0e96d998e admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:30.991251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4b0f2-76ed-465c-8a1e-48c0e96d998e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:30.991564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb4b0f2-76ed-465c-8a1e-48c0e96d998e admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:30.992438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb4b0f2-76ed-465c-8a1e-48c0e96d998e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:30.994236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb4b0f2-76ed-465c-8a1e-48c0e96d998e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:30.996566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aac2207-fae8-4a6e-8a4c-cfca7d4fd0b1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.003333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3edf2c2-bd63-4401-8f18-24332daddf5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.003697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edf2c2-bd63-4401-8f18-24332daddf5a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.003967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edf2c2-bd63-4401-8f18-24332daddf5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.004239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3edf2c2-bd63-4401-8f18-24332daddf5a admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.005082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3edf2c2-bd63-4401-8f18-24332daddf5a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.008573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3edf2c2-bd63-4401-8f18-24332daddf5a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.009694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 796/1596] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:31.011539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d64a26-6303-40bb-8310-1fc21fbad8f6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.011919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d64a26-6303-40bb-8310-1fc21fbad8f6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.012332 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59d64a26-6303-40bb-8310-1fc21fbad8f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.022529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 801/1597] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:30 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:31.025086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa09cf1-330e-4a7f-b5c1-6be305862043 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.025367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa09cf1-330e-4a7f-b5c1-6be305862043 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.025639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa09cf1-330e-4a7f-b5c1-6be305862043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.048774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-afa09cf1-330e-4a7f-b5c1-6be305862043 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:31.049956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-afa09cf1-330e-4a7f-b5c1-6be305862043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:31.052411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-59d64a26-6303-40bb-8310-1fc21fbad8f6 None None] MFA Rules not processed for user `1202a5b40a1d4c52bf5a8fe965f5f35e`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:31.053877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-59d64a26-6303-40bb-8310-1fc21fbad8f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:31.061763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 802/1598] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:31.062679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa09cf1-330e-4a7f-b5c1-6be305862043 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.073452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa09cf1-330e-4a7f-b5c1-6be305862043 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.079348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.079964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.080288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.080651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.081355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.083170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.097684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 803/1599] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:31 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:31.098946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.112707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ec4c9fb-6025-4f0c-83d4-7d2ffa9ba8d1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.117902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.118543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.118953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.119216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.119895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.121486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.133850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 804/1600] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:31 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:31.134868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.148734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e92c9f-6ea5-4efc-96ce-828d46c85f29 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.154507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.155254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.155531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.155799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.156442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.156870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 797/1601] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:31.157715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d64a26-6303-40bb-8310-1fc21fbad8f6 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.161473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.169371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59d64a26-6303-40bb-8310-1fc21fbad8f6 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.174341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.175136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.175505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.176163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] PATH_INFO: `/v3/projects/8f88658c804e40229386a559fd2a1fa9/users/0df60784a8b3468ca19ec2cee1d929e5/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.176284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 805/1602] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:31 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:31.177080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.186256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:31.189268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29c0ab-dbe9-41db-add4-7381f92d8a60 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.195139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.196225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.196541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.196861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8f88658c804e40229386a559fd2a1fa9, user_id=0df60784a8b3468ca19ec2cee1d929e5, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.196998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] PATH_INFO: `/v3/projects/9006f0bb109c4c96a387ff5c314f236e/users/a6111a16f95d4bc58f84d9f60f703d80/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.200874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.218928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9006f0bb109c4c96a387ff5c314f236e, user_id=a6111a16f95d4bc58f84d9f60f703d80, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.221420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 798/1603] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:31 2022] PUT /identity/v3/projects/8f88658c804e40229386a559fd2a1fa9/users/0df60784a8b3468ca19ec2cee1d929e5/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:31.223299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.225017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:31.225304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.244231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 806/1604] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:31 2022] PUT /identity/v3/projects/9006f0bb109c4c96a387ff5c314f236e/users/a6111a16f95d4bc58f84d9f60f703d80/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:31.245716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.297019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e8e00a-874d-4baf-aaf4-0cabb47d3286 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.327505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82c2ca8-1511-413c-9ea2-11d406bdd525 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.333708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.335172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.335654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.336088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] PATH_INFO: `/v3/projects/bf12aa8cbe4f41cd94cebc15164ef2fc/users/52d8eca1729b418e94e0359adca4f5af/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.342416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:31.362898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=bf12aa8cbe4f41cd94cebc15164ef2fc, user_id=52d8eca1729b418e94e0359adca4f5af, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.365822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.368621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.369629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.370189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.370761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.384219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.386867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.388086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 807/1605] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:31 2022] PUT /identity/v3/projects/bf12aa8cbe4f41cd94cebc15164ef2fc/users/52d8eca1729b418e94e0359adca4f5af/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 144 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:31.399947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.411432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 799/1606] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:31.413041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.437249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e12ee2-4d5a-438d-9412-1ab424fb3984 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.485994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cbe9c28-5c1e-4d3a-b454-fda862aff3ed admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.491735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.493216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.493697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.494074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] PATH_INFO: `/v3/projects/ad99e285720d41988cf2cf8f8978c313/users/d0ed4854e0d449e58d09487fe8eedc2c/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.514600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad99e285720d41988cf2cf8f8978c313, user_id=d0ed4854e0d449e58d09487fe8eedc2c, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.517877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.543677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 808/1607] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:31 2022] PUT /identity/v3/projects/ad99e285720d41988cf2cf8f8978c313/users/d0ed4854e0d449e58d09487fe8eedc2c/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:31.544877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:31.580735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea22adfe-fe2a-449b-b3ea-2c444347c2e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.581286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22adfe-fe2a-449b-b3ea-2c444347c2e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.582063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22adfe-fe2a-449b-b3ea-2c444347c2e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.582345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea22adfe-fe2a-449b-b3ea-2c444347c2e8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.582994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22adfe-fe2a-449b-b3ea-2c444347c2e8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.585348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea22adfe-fe2a-449b-b3ea-2c444347c2e8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.633970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 800/1608] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/projects => generated 371 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:31.635095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd904b-7615-4339-bf4e-ec862755df64 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.635371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd904b-7615-4339-bf4e-ec862755df64 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.635638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddd904b-7615-4339-bf4e-ec862755df64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.657385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6680d1-1dba-4771-bdb2-f5dafe17acbd admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:31.671180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8ddd904b-7615-4339-bf4e-ec862755df64 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:31.672621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8ddd904b-7615-4339-bf4e-ec862755df64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:31.713842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:31.760986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87fd5e03-f53a-476a-addc-3bc5d567aff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:31.761865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd5e03-f53a-476a-addc-3bc5d567aff0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.762261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd5e03-f53a-476a-addc-3bc5d567aff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.762576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87fd5e03-f53a-476a-addc-3bc5d567aff0 admin admin] PATH_INFO: `/v3/projects/6c8694988d8e4d72a26790d23a332f2c/users/e44e3c8ac0244d598fda750643355e1e/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.797641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fd5e03-f53a-476a-addc-3bc5d567aff0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c8694988d8e4d72a26790d23a332f2c, user_id=e44e3c8ac0244d598fda750643355e1e, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:31.800911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87fd5e03-f53a-476a-addc-3bc5d567aff0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:31.810241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:31.824563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 801/1609] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:31.826870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29139ea0-2e07-4b14-9d56-f4a92a116243 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.827392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29139ea0-2e07-4b14-9d56-f4a92a116243 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.827951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29139ea0-2e07-4b14-9d56-f4a92a116243 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.829652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 809/1610] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:31 2022] PUT /identity/v3/projects/6c8694988d8e4d72a26790d23a332f2c/users/e44e3c8ac0244d598fda750643355e1e/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:31.831076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3a53a0-2b98-48e6-af11-c37549f6b8d6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.831183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3a53a0-2b98-48e6-af11-c37549f6b8d6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.831420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3a53a0-2b98-48e6-af11-c37549f6b8d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.856847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-29139ea0-2e07-4b14-9d56-f4a92a116243 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:31.858544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-29139ea0-2e07-4b14-9d56-f4a92a116243 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:31.888317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3d3a53a0-2b98-48e6-af11-c37549f6b8d6 None None] MFA Rules not processed for user `52d8eca1729b418e94e0359adca4f5af`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:31.889524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3d3a53a0-2b98-48e6-af11-c37549f6b8d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:31.933205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 802/1611] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:31.934721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2a4b78-ed4b-48fe-bdbd-fdc002f1e5fb None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:31.935013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2a4b78-ed4b-48fe-bdbd-fdc002f1e5fb None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:31.935470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee2a4b78-ed4b-48fe-bdbd-fdc002f1e5fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:31.961563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ee2a4b78-ed4b-48fe-bdbd-fdc002f1e5fb None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:31.962937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ee2a4b78-ed4b-48fe-bdbd-fdc002f1e5fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:31.981065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 803/1612] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:31.981634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2a4b78-ed4b-48fe-bdbd-fdc002f1e5fb None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.010804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee2a4b78-ed4b-48fe-bdbd-fdc002f1e5fb None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.019316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.020320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.020938 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.021306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 810/1613] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:32.021900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.022676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a9706-d44d-4021-9260-10cd528fc135 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.022962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a9706-d44d-4021-9260-10cd528fc135 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.023069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.117446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-831a9706-d44d-4021-9260-10cd528fc135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.120804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.121652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-831a9706-d44d-4021-9260-10cd528fc135 None None] MFA Rules not processed for user `e44e3c8ac0244d598fda750643355e1e`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:32.121652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-831a9706-d44d-4021-9260-10cd528fc135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:32.131395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 804/1614] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:31 2022] POST /identity/v3/users => generated 292 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:32.131395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.238951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52b7cf00-e53b-4e9b-b310-2153648410f7 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.241524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 811/1615] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:32 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:32.246264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.246264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.246264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.246264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.247717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831a9706-d44d-4021-9260-10cd528fc135 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.260755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.262263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.274549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 805/1616] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:32.275414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.276020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-831a9706-d44d-4021-9260-10cd528fc135 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.283610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8564a178-366b-4b9d-8232-3c46d29d3b81 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.286991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.287314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.287990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.288193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.288542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.290662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.291894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.293333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.303302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.304751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.312949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.317472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.328032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 806/1617] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:32.329238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.329976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 812/1618] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:32.332056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.343076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8109d603-c200-48c1-81e0-49c980e517fb service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.344017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03384964-30ad-446c-95e8-12794ad05651 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.350318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.350958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.351236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.352406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.352646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.353336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.353458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.353847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.354239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.355315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.355984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.359163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.371518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 813/1619] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:32.372914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.386277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11a0a771-a8c9-449a-920c-bf3e6e529304 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.393334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.394224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.394578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.394919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.395674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.399488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.413550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 807/1620] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:32 2022] POST /identity/v3/projects => generated 387 bytes in 82 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:32.413550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.442342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:32.450247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 814/1621] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:32 2022] POST /identity/v3/projects => generated 357 bytes in 78 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:32.451512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.502947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:32.515268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e072773-03b6-4a02-b771-0a7066b1ae1b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.623168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9879a2-04c0-40e2-92b3-305e0661f60f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.638962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.639614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.639896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.640175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.642222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.644258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.648915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.649605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.649882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.650735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] PATH_INFO: `/v3/projects/64c70e18c9a1496cbb96e0087254c5db/users/1610a4fe00884d379f7a2f9111d8977d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.686638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64c70e18c9a1496cbb96e0087254c5db, user_id=1610a4fe00884d379f7a2f9111d8977d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.691757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.709078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 815/1622] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:32 2022] POST /identity/v3/users => generated 300 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:32.710453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.742123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5ce150-f36f-43e4-b1b5-7dd212f186c6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.750101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 808/1623] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:32 2022] PUT /identity/v3/projects/64c70e18c9a1496cbb96e0087254c5db/users/1610a4fe00884d379f7a2f9111d8977d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:32.751339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.895359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.896556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.897134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.897660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.900296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.903941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-196fa7fc-b948-4c9a-827f-8d22152cf010 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.910923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.916660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.919030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.919310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.921917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.922726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.926747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.950229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 816/1624] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:32 2022] POST /identity/v3/users => generated 285 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:32.951701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.960531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 809/1625] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/roles => generated 1420 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:32.961557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:32.969446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0deeb566-0df0-419c-a243-63f1571661b9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:32.978761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:32.980182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:32.980766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:32.981814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:32.983324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:32.986688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:32.990674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a81776a-dc51-4aee-9455-d72fa7242fd6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.003593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.004599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.006545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 817/1626] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:33.007757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.009788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.012910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.013638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.015670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.023051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-470acc61-d04c-4e1d-881f-6f50e271adc9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.034477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.035222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.035550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.036832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] PATH_INFO: `/v3/projects/2f86e7cfb31b42ee8b6198d12767632c/users/96950dc92fdb4bd5a25808951f65b4a2/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.047665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 810/1627] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:32 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:33.050042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.063175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f86e7cfb31b42ee8b6198d12767632c, user_id=96950dc92fdb4bd5a25808951f65b4a2, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.069883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.078483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37dbdb07-bbc1-4e56-922b-c526cf55f822 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.091482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.093557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.093845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.094483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] PATH_INFO: `/v3/projects/64c70e18c9a1496cbb96e0087254c5db/users/1610a4fe00884d379f7a2f9111d8977d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.096384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 818/1628] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:33 2022] PUT /identity/v3/projects/2f86e7cfb31b42ee8b6198d12767632c/users/96950dc92fdb4bd5a25808951f65b4a2/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:33.097350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.100264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] RBAC: Authorizing `identity:create_grant(project_id=64c70e18c9a1496cbb96e0087254c5db, user_id=1610a4fe00884d379f7a2f9111d8977d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.108018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.136681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 811/1629] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:33 2022] PUT /identity/v3/projects/64c70e18c9a1496cbb96e0087254c5db/users/1610a4fe00884d379f7a2f9111d8977d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 88 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:33.137561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.180312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:33.257128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762838e2-459d-4970-842e-ecae2db06672 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.280584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.281321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.281598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.281871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] PATH_INFO: `/v3/projects/8e763b80ea264b42a88d1507fdc35830/users/5d28e1984ad44d53af3df1b99c4678ad/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.284696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6959eb97-067b-4dd1-9027-e67c27eea483 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.292557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a436f1-eb9d-4315-a674-f9dc92f15c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.294455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a436f1-eb9d-4315-a674-f9dc92f15c12 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.294777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a436f1-eb9d-4315-a674-f9dc92f15c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.295056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a436f1-eb9d-4315-a674-f9dc92f15c12 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.295877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a436f1-eb9d-4315-a674-f9dc92f15c12 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.298684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a436f1-eb9d-4315-a674-f9dc92f15c12 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.322042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 812/1630] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:33 2022] GET /identity/v3/roles => generated 1420 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:33.323520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e763b80ea264b42a88d1507fdc35830, user_id=5d28e1984ad44d53af3df1b99c4678ad, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.323838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f644daab-ca73-419e-8a6b-ac5dcbbed9fa None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.324182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f644daab-ca73-419e-8a6b-ac5dcbbed9fa None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.324493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f644daab-ca73-419e-8a6b-ac5dcbbed9fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.327114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.344552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:33.363046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 819/1631] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:33 2022] PUT /identity/v3/projects/8e763b80ea264b42a88d1507fdc35830/users/5d28e1984ad44d53af3df1b99c4678ad/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:33.363875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.392534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f644daab-ca73-419e-8a6b-ac5dcbbed9fa None None] MFA Rules not processed for user `1610a4fe00884d379f7a2f9111d8977d`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:33.394285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f644daab-ca73-419e-8a6b-ac5dcbbed9fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:33.449077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ab3564f-3f8b-4887-b715-6ec6bb061ec0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.456153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.456977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.457343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.458121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] PATH_INFO: `/v3/projects/2f86e7cfb31b42ee8b6198d12767632c/users/96950dc92fdb4bd5a25808951f65b4a2/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.460924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] RBAC: Authorizing `identity:create_grant(project_id=2f86e7cfb31b42ee8b6198d12767632c, user_id=96950dc92fdb4bd5a25808951f65b4a2, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.465049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.479147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 820/1632] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:33 2022] PUT /identity/v3/projects/2f86e7cfb31b42ee8b6198d12767632c/users/96950dc92fdb4bd5a25808951f65b4a2/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:33.480146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.493908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 813/1633] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:33 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:33.495554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e8b3b-ca97-4ac5-9cef-ae228fc850a2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.496051 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e8b3b-ca97-4ac5-9cef-ae228fc850a2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.496531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de1e8b3b-ca97-4ac5-9cef-ae228fc850a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.531985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-de1e8b3b-ca97-4ac5-9cef-ae228fc850a2 None None] MFA Rules not processed for user `96950dc92fdb4bd5a25808951f65b4a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:33.534138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-de1e8b3b-ca97-4ac5-9cef-ae228fc850a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:33.582349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92a417e1-c1a3-469c-aa80-3c64b55e183d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.592444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.593935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.594956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.595383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.596614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.599856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.623033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 821/1634] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:33 2022] GET /identity/v3/roles => generated 1420 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:33.623785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.633622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 814/1635] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:33 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:33.633622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e8b3b-ca97-4ac5-9cef-ae228fc850a2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.642855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de1e8b3b-ca97-4ac5-9cef-ae228fc850a2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.649839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.650508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.650846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.651187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] PATH_INFO: `/v3/projects/8e763b80ea264b42a88d1507fdc35830/users/5d28e1984ad44d53af3df1b99c4678ad/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.653413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=8e763b80ea264b42a88d1507fdc35830, user_id=5d28e1984ad44d53af3df1b99c4678ad, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.656341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.669572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 815/1636] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:33 2022] PUT /identity/v3/projects/8e763b80ea264b42a88d1507fdc35830/users/5d28e1984ad44d53af3df1b99c4678ad/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:33.670587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:33.686117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:33.753629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b4f172-f2af-4845-9329-c782338941bb admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.757822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.758150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20d8c814-5d2e-4f06-9b4a-3d21001983e1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:33.758818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.759371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.759631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.763070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccfd5294-45d6-4c2f-bbe1-f61b93d52e1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:33.763698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfd5294-45d6-4c2f-bbe1-f61b93d52e1f service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.763974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfd5294-45d6-4c2f-bbe1-f61b93d52e1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.764532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccfd5294-45d6-4c2f-bbe1-f61b93d52e1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.823839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:33.857389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfd5294-45d6-4c2f-bbe1-f61b93d52e1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.859196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccfd5294-45d6-4c2f-bbe1-f61b93d52e1f service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.867372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:33.871711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:33.873415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 822/1637] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:33 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 250 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:33.874800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ecd736-8a18-401e-84e3-7f0be96c9ad8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:33.875092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ecd736-8a18-401e-84e3-7f0be96c9ad8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:33.875366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ecd736-8a18-401e-84e3-7f0be96c9ad8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:33.883918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 816/1638] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:33 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 214 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:33.912611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a2ecd736-8a18-401e-84e3-7f0be96c9ad8 None None] MFA Rules not processed for user `5d28e1984ad44d53af3df1b99c4678ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:33.914276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a2ecd736-8a18-401e-84e3-7f0be96c9ad8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:33.925950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.060241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c94d5cab-00be-4547-aad7-22be282b5ac6 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.081570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.082297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.082678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.083485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] PATH_INFO: `/v3/users/52d8eca1729b418e94e0359adca4f5af` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.086515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52d8eca1729b418e94e0359adca4f5af)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.090214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: WARNING py.warnings [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:12:34.090214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: warnings.warn(msg) Sep 29 02:12:34.090214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: Sep 29 02:12:34.091502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.105181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 823/1639] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:33 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:34.105181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ecd736-8a18-401e-84e3-7f0be96c9ad8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.128361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ecd736-8a18-401e-84e3-7f0be96c9ad8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.141853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.143107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.143320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.143586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.247827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52d8eca1729b418e94e0359adca4f5af'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.280568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52d8eca1729b418e94e0359adca4f5af'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.281321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52d8eca1729b418e94e0359adca4f5af'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.281973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52d8eca1729b418e94e0359adca4f5af'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.298624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.301297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.311856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b3789449-3d2d-4607-b206-2383c486deb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52d8eca1729b418e94e0359adca4f5af'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:34.313624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:34.330392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 824/1640] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:34 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:34.331398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.351256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dd725ba-fb04-4248-99d4-ad1057cbbe29 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.361014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.361707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.361929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.362232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.368393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.373564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 817/1641] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:33 2022] DELETE /identity/v3/users/52d8eca1729b418e94e0359adca4f5af => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:34.377841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.378353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.378850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.398024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.399602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.428029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:34.433514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.437097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 825/1642] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:34 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:34.547230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 818/1643] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:34.597160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.599254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.617520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f586e330-0ed9-464e-b1fe-979dd0649e17 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.693869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:34.713218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.713840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.714129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.714399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.716627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.730138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:34.732617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:34.864629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7e2ccd3-1603-40f2-a476-a6a0a8547351 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:34.873911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:34.874595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:34.874875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:34.875146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:34.902699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.904819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.937944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 819/1644] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:34.963081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:34.964666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:34.986390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 826/1645] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:34 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 390 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:36.189029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:36.190830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:36.219386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b19a058-5ed9-4689-8e5e-124e1618e969 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:36.248860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:36.249195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:36.249298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:36.249538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] PATH_INFO: `/v3/projects/bf12aa8cbe4f41cd94cebc15164ef2fc` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:36.251557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] RBAC: Authorizing `identity:delete_project(project_id=bf12aa8cbe4f41cd94cebc15164ef2fc)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:36.253545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:36.369950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bf12aa8cbe4f41cd94cebc15164ef2fc', 'request_id': 'req-90ccb1c6-32df-41f9-a2c2-009749ab156a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.395082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bf12aa8cbe4f41cd94cebc15164ef2fc', 'request_id': 'req-90ccb1c6-32df-41f9-a2c2-009749ab156a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.405762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bf12aa8cbe4f41cd94cebc15164ef2fc', 'request_id': 'req-90ccb1c6-32df-41f9-a2c2-009749ab156a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.430170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] The token cache is being invalidate because project bf12aa8cbe4f41cd94cebc15164ef2fc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:36.432123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.433533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.440690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 820/1646] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:36 2022] DELETE /identity/v3/projects/bf12aa8cbe4f41cd94cebc15164ef2fc => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:36.441852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:36.456521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7638433-c5d7-492e-934a-8a5fd1f3be05 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:36.660868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:36.661528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:36.661811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:36.662116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] PATH_INFO: `/v3/users/96950dc92fdb4bd5a25808951f65b4a2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:36.666201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] RBAC: Authorizing `identity:delete_user(user_id=96950dc92fdb4bd5a25808951f65b4a2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:36.669825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:36.775741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '96950dc92fdb4bd5a25808951f65b4a2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.791074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '96950dc92fdb4bd5a25808951f65b4a2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.791814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '96950dc92fdb4bd5a25808951f65b4a2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.792448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '96950dc92fdb4bd5a25808951f65b4a2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.818179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bb65ae64-2a66-430f-9a7b-f82299d2e59f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '96950dc92fdb4bd5a25808951f65b4a2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:36.857160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 827/1647] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:36 2022] DELETE /identity/v3/users/96950dc92fdb4bd5a25808951f65b4a2 => generated 0 bytes in 416 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:38.841863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:38.844411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:38.874751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ccb1c6-32df-41f9-a2c2-009749ab156a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:38.893099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:38.994218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:38.995356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:38.995696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:38.996480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] PATH_INFO: `/v3/projects/2f86e7cfb31b42ee8b6198d12767632c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:38.999530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2f86e7cfb31b42ee8b6198d12767632c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:39.003722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:39.108623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2f86e7cfb31b42ee8b6198d12767632c', 'request_id': 'req-569928ce-6ee3-457b-b58a-a652d684d764'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:39.121038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2f86e7cfb31b42ee8b6198d12767632c', 'request_id': 'req-569928ce-6ee3-457b-b58a-a652d684d764'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:39.145493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2f86e7cfb31b42ee8b6198d12767632c', 'request_id': 'req-569928ce-6ee3-457b-b58a-a652d684d764'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:39.163956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] The token cache is being invalidate because project 2f86e7cfb31b42ee8b6198d12767632c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:39.164146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:39.168706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-569928ce-6ee3-457b-b58a-a652d684d764 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:39.175915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 821/1648] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:38 2022] DELETE /identity/v3/projects/2f86e7cfb31b42ee8b6198d12767632c => generated 0 bytes in 329 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:39.176118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d7568-77cb-4bbc-ae4d-2d36a45df9b1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.176631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d7568-77cb-4bbc-ae4d-2d36a45df9b1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.176718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2d7568-77cb-4bbc-ae4d-2d36a45df9b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.215540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-df2d7568-77cb-4bbc-ae4d-2d36a45df9b1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:39.217061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-df2d7568-77cb-4bbc-ae4d-2d36a45df9b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:39.246854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:39.380846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 828/1649] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:39.384596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb5e57-5f19-4290-8232-6a5bb98aab6a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.385095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb5e57-5f19-4290-8232-6a5bb98aab6a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.385585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23eb5e57-5f19-4290-8232-6a5bb98aab6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.435241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-23eb5e57-5f19-4290-8232-6a5bb98aab6a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:39.439272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-23eb5e57-5f19-4290-8232-6a5bb98aab6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:39.475342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2d7568-77cb-4bbc-ae4d-2d36a45df9b1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:39.476418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 822/1650] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:39.487718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2d7568-77cb-4bbc-ae4d-2d36a45df9b1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:39.500326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:39.500970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.501252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.501520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.502227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:39.502530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:39.510716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:39.532710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 829/1651] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:39.536942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eb5e57-5f19-4290-8232-6a5bb98aab6a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:39.572286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23eb5e57-5f19-4290-8232-6a5bb98aab6a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:39.577612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:39.578398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.578805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.579194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.582855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:39.584997 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:39.683564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 823/1652] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:39 2022] POST /identity/v3/projects => generated 391 bytes in 144 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:39.684777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:39.696895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d3b4f9-180d-4677-add9-2dc22325a0b0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:39.859835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:39.860654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.861049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.867720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.869078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:39.872677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:39.920806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 830/1653] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:39 2022] POST /identity/v3/users => generated 296 bytes in 236 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:39.925226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:39.946961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcadd552-427c-4b70-ab3c-cd7309ed0fbb admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:39.964276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:39.965288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:39.965713 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:39.966149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:39.966971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:39.969294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:39.997168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 824/1654] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 72 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:40.001133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:40.019728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24aac93-2559-4bef-a487-de5715176bf2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:40.038941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f520aed2-1c97-4876-acbb-16e3514dec58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:40.039919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f520aed2-1c97-4876-acbb-16e3514dec58 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.043720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f520aed2-1c97-4876-acbb-16e3514dec58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.044107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f520aed2-1c97-4876-acbb-16e3514dec58 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:40.044924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f520aed2-1c97-4876-acbb-16e3514dec58 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:40.048871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f520aed2-1c97-4876-acbb-16e3514dec58 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:40.078241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 831/1655] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:39 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:40.082298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:40.095331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3903c56-4167-4acf-90e3-a94158bcb14a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:40.102532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:40.103898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.104465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.104928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] PATH_INFO: `/v3/projects/eed46241faa14306b5a4661fbd169c25/users/e4e34d9e4e34497382e0a5c791ca0e65/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:40.172427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eed46241faa14306b5a4661fbd169c25, user_id=e4e34d9e4e34497382e0a5c791ca0e65, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:40.178547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:40.214233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9378b0fa-ee21-4ed1-8f41-42f0e09b1c42 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.214779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9378b0fa-ee21-4ed1-8f41-42f0e09b1c42 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.214779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9378b0fa-ee21-4ed1-8f41-42f0e09b1c42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:40.222074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 825/1656] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:40 2022] PUT /identity/v3/projects/eed46241faa14306b5a4661fbd169c25/users/e4e34d9e4e34497382e0a5c791ca0e65/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:40.292051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9378b0fa-ee21-4ed1-8f41-42f0e09b1c42 None None] MFA Rules not processed for user `e4e34d9e4e34497382e0a5c791ca0e65`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:40.293294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9378b0fa-ee21-4ed1-8f41-42f0e09b1c42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:40.508278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 832/1657] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:40 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 296 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:40.512928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:40.583928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:40.661701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7901ac-8a88-4edd-af00-d310d613eb73 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:40.676569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:40.677223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.677503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.677776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:40.678515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:40.682813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:40.746379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 826/1658] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:40 2022] POST /identity/v3/projects => generated 391 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:40.747577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9378b0fa-ee21-4ed1-8f41-42f0e09b1c42 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:40.759788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9378b0fa-ee21-4ed1-8f41-42f0e09b1c42 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:40.772891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:40.851202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:40.852305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.852914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.866740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:40.867844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:40.874608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:40.914339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:40.915574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 833/1659] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:40 2022] POST /identity/v3/users => generated 302 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:40.932697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-799e5927-818b-4492-b552-dc71373e6b4f admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:40.940847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:40.944877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.945369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.945777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:40.946804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:40.948981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:40.969580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:40.982932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 827/1660] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:40 2022] GET /identity/v3/roles => generated 1420 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:40.986809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2bd9e3b-1744-40eb-8584-902a361c86bb admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:40.994330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:40.995300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:40.995714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:40.996150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] PATH_INFO: `/v3/projects/9d2fdb2758d241c38ce51f2651c6c937/users/2179461e07bd4ef6a120702ba6d07c8d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.016719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d2fdb2758d241c38ce51f2651c6c937, user_id=2179461e07bd4ef6a120702ba6d07c8d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:41.020591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:41.046751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:41.047186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 834/1661] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:40 2022] PUT /identity/v3/projects/9d2fdb2758d241c38ce51f2651c6c937/users/2179461e07bd4ef6a120702ba6d07c8d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:41.061533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dbc133b-b15a-4a56-bbfc-347d32b260c8 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:41.271169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d289a99-951c-4b7a-92fb-2b9ffbf298ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:41.276582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d289a99-951c-4b7a-92fb-2b9ffbf298ab admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.277891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d289a99-951c-4b7a-92fb-2b9ffbf298ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.277891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d289a99-951c-4b7a-92fb-2b9ffbf298ab admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.278486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d289a99-951c-4b7a-92fb-2b9ffbf298ab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:41.289505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d289a99-951c-4b7a-92fb-2b9ffbf298ab admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:41.324505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:41.335621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b87340a-cfa3-4843-b64a-540ac56b8f8d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:41.340303 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 828/1662] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:41 2022] GET /identity/v3/roles => generated 1420 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:41.354543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:41.355229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.355501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.355768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] PATH_INFO: `/v3/projects/9d2fdb2758d241c38ce51f2651c6c937/users/2179461e07bd4ef6a120702ba6d07c8d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.360330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9d2fdb2758d241c38ce51f2651c6c937, user_id=2179461e07bd4ef6a120702ba6d07c8d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:41.368562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:41.402662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 835/1663] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:41 2022] PUT /identity/v3/projects/9d2fdb2758d241c38ce51f2651c6c937/users/2179461e07bd4ef6a120702ba6d07c8d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:41.416015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.416314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.416741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.469214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] MFA Rules not processed for user `2179461e07bd4ef6a120702ba6d07c8d`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:41.470619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:41.684139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 829/1664] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:41 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:41.698767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:41.701153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:41.780355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fa24bf8-15af-4b35-9b14-7b3fbea40116 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:41.785511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7799c733-eda0-49c7-9ec4-15da56881679 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:41.786241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799c733-eda0-49c7-9ec4-15da56881679 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.786709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799c733-eda0-49c7-9ec4-15da56881679 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.786814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7799c733-eda0-49c7-9ec4-15da56881679 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.797546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7799c733-eda0-49c7-9ec4-15da56881679 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:41.799310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7799c733-eda0-49c7-9ec4-15da56881679 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:41.816833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 836/1665] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:41 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:41.897431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:41.899437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:41.915193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe1f84b-a74e-43d5-9ee5-7c2be87c56cd None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:41.923648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89f12d6-a82b-4323-b6c6-09de60583d9c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:41.925254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89f12d6-a82b-4323-b6c6-09de60583d9c service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:41.925682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89f12d6-a82b-4323-b6c6-09de60583d9c service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:41.926188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89f12d6-a82b-4323-b6c6-09de60583d9c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:41.994556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:42.049590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89f12d6-a82b-4323-b6c6-09de60583d9c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:42.051403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c89f12d6-a82b-4323-b6c6-09de60583d9c service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:42.069100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 830/1666] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:41 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:43.009831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ee6d1e-dc2f-4957-8513-6d7e420fb14b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:43.010366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ee6d1e-dc2f-4957-8513-6d7e420fb14b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:43.010448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96ee6d1e-dc2f-4957-8513-6d7e420fb14b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:43.045617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-96ee6d1e-dc2f-4957-8513-6d7e420fb14b None None] MFA Rules not processed for user `a6111a16f95d4bc58f84d9f60f703d80`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:43.047240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-96ee6d1e-dc2f-4957-8513-6d7e420fb14b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:43.145899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:43.150542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 837/1667] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:43 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:43.202748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a23e74-4f8d-4a91-9fa2-0e8d8f9e95af None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:43.203247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a23e74-4f8d-4a91-9fa2-0e8d8f9e95af None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:43.203551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a23e74-4f8d-4a91-9fa2-0e8d8f9e95af None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:43.205191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 831/1668] 10.222.0.61 () {54 vars in 961 bytes} [Thu Sep 29 02:12:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:43.214787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb493f5-0640-4370-bb47-4e93838ac137 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:43.217994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb493f5-0640-4370-bb47-4e93838ac137 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:43.218640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fb493f5-0640-4370-bb47-4e93838ac137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:43.298460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4fb493f5-0640-4370-bb47-4e93838ac137 None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:43.300939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4fb493f5-0640-4370-bb47-4e93838ac137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:43.408698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a23e74-4f8d-4a91-9fa2-0e8d8f9e95af None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:43.420270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 838/1669] 10.222.0.61 () {58 vars in 1058 bytes} [Thu Sep 29 02:12:43 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:43.428394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5a23e74-4f8d-4a91-9fa2-0e8d8f9e95af None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:43.438324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd53de4e-4ede-4917-96f1-c3ec9b7eca0d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:43.439150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd53de4e-4ede-4917-96f1-c3ec9b7eca0d service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:43.440297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd53de4e-4ede-4917-96f1-c3ec9b7eca0d service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:43.440773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd53de4e-4ede-4917-96f1-c3ec9b7eca0d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:43.451847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd53de4e-4ede-4917-96f1-c3ec9b7eca0d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:43.455244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd53de4e-4ede-4917-96f1-c3ec9b7eca0d service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:43.488947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 832/1670] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:43 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 81 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:45.706037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017f83f-c581-498e-8b98-846c4683dd10 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:45.707473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017f83f-c581-498e-8b98-846c4683dd10 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:45.707981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5017f83f-c581-498e-8b98-846c4683dd10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:45.804240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5017f83f-c581-498e-8b98-846c4683dd10 None None] MFA Rules not processed for user `3b624208bb94427791219ffcca0efc41`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:45.809063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5017f83f-c581-498e-8b98-846c4683dd10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.000092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c271d5e-449f-4bbb-a8b9-f5c0b8b591b2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.000564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c271d5e-449f-4bbb-a8b9-f5c0b8b591b2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.000884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c271d5e-449f-4bbb-a8b9-f5c0b8b591b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.046229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 839/1671] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 341 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:46.058211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788add1c-f6bf-49b8-ab70-099bdbf2f8ee None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.058507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788add1c-f6bf-49b8-ab70-099bdbf2f8ee None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.058785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788add1c-f6bf-49b8-ab70-099bdbf2f8ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.131473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-788add1c-f6bf-49b8-ab70-099bdbf2f8ee None None] MFA Rules not processed for user `3b624208bb94427791219ffcca0efc41`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:46.132717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-788add1c-f6bf-49b8-ab70-099bdbf2f8ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.133200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0c271d5e-449f-4bbb-a8b9-f5c0b8b591b2 None None] MFA Rules not processed for user `480c3f60b5e14b5f93e5dc921694aee4`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:46.135311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0c271d5e-449f-4bbb-a8b9-f5c0b8b591b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.162142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 840/1672] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:46 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:46.206828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2a42-dc27-46ac-893e-6097bfb3e53e None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.207100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2a42-dc27-46ac-893e-6097bfb3e53e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.207370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fb2a42-dc27-46ac-893e-6097bfb3e53e None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.209025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 841/1673] 10.222.0.61 () {54 vars in 961 bytes} [Thu Sep 29 02:12:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:46.245349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994b6de2-b49f-4e33-b036-a80b78b5ce53 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.245621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994b6de2-b49f-4e33-b036-a80b78b5ce53 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.245995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994b6de2-b49f-4e33-b036-a80b78b5ce53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.295041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-994b6de2-b49f-4e33-b036-a80b78b5ce53 None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:46.297819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-994b6de2-b49f-4e33-b036-a80b78b5ce53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:46.362972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 833/1674] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:45 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 365 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:46.370542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 842/1675] 10.222.0.61 () {58 vars in 1058 bytes} [Thu Sep 29 02:12:46 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:46.378825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c271d5e-449f-4bbb-a8b9-f5c0b8b591b2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.385875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994b6de2-b49f-4e33-b036-a80b78b5ce53 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:46.424248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c271d5e-449f-4bbb-a8b9-f5c0b8b591b2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.425407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994b6de2-b49f-4e33-b036-a80b78b5ce53 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:46.446979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23da6921-8acc-4793-a867-206773b36aff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.449636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23da6921-8acc-4793-a867-206773b36aff service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.449955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23da6921-8acc-4793-a867-206773b36aff service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.450265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23da6921-8acc-4793-a867-206773b36aff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.466649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23da6921-8acc-4793-a867-206773b36aff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.467459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80acee4-715f-4909-86a5-bd4698e7ca4f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:46.468213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80acee4-715f-4909-86a5-bd4698e7ca4f service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.468360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23da6921-8acc-4793-a867-206773b36aff service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.471121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80acee4-715f-4909-86a5-bd4698e7ca4f service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.472738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80acee4-715f-4909-86a5-bd4698e7ca4f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:46.492696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 834/1676] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:46 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:46.512281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80acee4-715f-4909-86a5-bd4698e7ca4f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:46.515062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80acee4-715f-4909-86a5-bd4698e7ca4f service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:46.549125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 843/1677] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:46 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:46.990111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885b132f-4da2-461e-a0da-5ca6be23bbba None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:46.990459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885b132f-4da2-461e-a0da-5ca6be23bbba None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:46.990730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-885b132f-4da2-461e-a0da-5ca6be23bbba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.040959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-885b132f-4da2-461e-a0da-5ca6be23bbba None None] MFA Rules not processed for user `a6111a16f95d4bc58f84d9f60f703d80`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:47.045153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-885b132f-4da2-461e-a0da-5ca6be23bbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.066304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 835/1678] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:46 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:47.134614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56701499-7cb3-4a02-b2ea-61562e438b29 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.134614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56701499-7cb3-4a02-b2ea-61562e438b29 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.134614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56701499-7cb3-4a02-b2ea-61562e438b29 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.134614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 844/1679] 10.222.0.61 () {54 vars in 912 bytes} [Thu Sep 29 02:12:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:47.146244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.146707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.147370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.192695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:47.196241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.221470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 836/1680] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:12:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:47.232576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f554fae-2445-48b9-9f73-a9b109b961d6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.232941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f554fae-2445-48b9-9f73-a9b109b961d6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.233221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f554fae-2445-48b9-9f73-a9b109b961d6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.235650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 845/1681] 10.222.0.61 () {54 vars in 912 bytes} [Thu Sep 29 02:12:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:47.241475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.244052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.261116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.262364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.262743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.314407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:47.319023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.335554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 846/1682] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:12:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:47.380183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:47.382959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:47.481544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e978f43-654e-4aca-b854-b835ac6706bc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.482704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:47.490878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfbe516-7bb9-465b-aef5-4719bd844c85 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.492133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfbe516-7bb9-465b-aef5-4719bd844c85 service glance] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.492500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfbe516-7bb9-465b-aef5-4719bd844c85 service glance] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.492860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfbe516-7bb9-465b-aef5-4719bd844c85 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.526449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfbe516-7bb9-465b-aef5-4719bd844c85 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.527726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-184894ce-e291-462e-8312-a0ed090fa4d5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:47.528450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfbe516-7bb9-465b-aef5-4719bd844c85 service glance] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.536988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:47.538681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cd1d05-b403-4d66-a032-536863b8e376 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:47.539352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cd1d05-b403-4d66-a032-536863b8e376 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:47.539636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cd1d05-b403-4d66-a032-536863b8e376 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:47.539927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cd1d05-b403-4d66-a032-536863b8e376 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:47.545225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 837/1683] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:47.564730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cd1d05-b403-4d66-a032-536863b8e376 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:47.566356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cd1d05-b403-4d66-a032-536863b8e376 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:47.595554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 847/1684] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 218 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:48.452247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af05b7-a072-4ef3-bb20-a8a04059c0ca None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.452696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af05b7-a072-4ef3-bb20-a8a04059c0ca None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.452784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0af05b7-a072-4ef3-bb20-a8a04059c0ca None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.454183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 838/1685] 10.222.0.61 () {54 vars in 912 bytes} [Thu Sep 29 02:12:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:48.461453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.461890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.462274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.580047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] MFA Rules not processed for user `cc2f041fe7194a3da199517b4cdbd265`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:48.581788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:48.768438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 848/1686] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:12:48 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 308 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:48.798350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.799366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d6e293-b9b5-46ea-96cf-a828c5b30bc3 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.799623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d6e293-b9b5-46ea-96cf-a828c5b30bc3 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.800598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d6e293-b9b5-46ea-96cf-a828c5b30bc3 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.801555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:48.809972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 839/1687] 10.222.0.61 () {54 vars in 912 bytes} [Thu Sep 29 02:12:48 2022] GET /identity => generated 270 bytes in 13 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:12:48.826407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.826683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.826961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.854207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7282d937-4f89-47c0-90c5-b01ee90aa4a7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:48.874483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] MFA Rules not processed for user `cc2f041fe7194a3da199517b4cdbd265`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:48.881094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:48.882273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:48.887603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:48.889164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:48.890839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:48.901585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 840/1688] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:12:48 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:48.931712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:48.942802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:48.959742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 849/1689] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:48.972246 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:48.973735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:48.998121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0016a19-7900-469d-8184-9e1c48422f91 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:49.014659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:49.015459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:49.015824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:49.016880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:49.031007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:49.032584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:49.058666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 841/1690] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:50.732889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:50.735012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:50.954106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-682edec5-e134-474f-95ea-8ca780f3365c service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:50.970552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6626ee-3aaf-46ac-b7c5-ac155beb2161 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:50.971184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6626ee-3aaf-46ac-b7c5-ac155beb2161 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:50.971825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6626ee-3aaf-46ac-b7c5-ac155beb2161 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:50.978630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6626ee-3aaf-46ac-b7c5-ac155beb2161 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:50.982299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ac6626ee-3aaf-46ac-b7c5-ac155beb2161 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:51.025838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6626ee-3aaf-46ac-b7c5-ac155beb2161 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:51.034980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6626ee-3aaf-46ac-b7c5-ac155beb2161 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:51.081921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 850/1691] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:50 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:52.533987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:52.827295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f4317e3-74ea-4c85-a992-9c9ad3f2cc8f service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:52.846599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:52.849841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:52.852638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:52.853994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] PATH_INFO: `/v3/users/e4e34d9e4e34497382e0a5c791ca0e65` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:52.860614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4e34d9e4e34497382e0a5c791ca0e65)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:52.877168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:53.080325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4e34d9e4e34497382e0a5c791ca0e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.113422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4e34d9e4e34497382e0a5c791ca0e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.115253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4e34d9e4e34497382e0a5c791ca0e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.116415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4e34d9e4e34497382e0a5c791ca0e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.140015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9a2f3ef3-7ca0-48b3-b6eb-148876e25b3e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4e34d9e4e34497382e0a5c791ca0e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:53.226033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51079a-bd9f-416e-84f0-d36855170f61 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.226915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51079a-bd9f-416e-84f0-d36855170f61 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.227404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f51079a-bd9f-416e-84f0-d36855170f61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.229337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 842/1692] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:52 2022] DELETE /identity/v3/users/e4e34d9e4e34497382e0a5c791ca0e65 => generated 0 bytes in 692 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:53.230919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.231324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.231737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.275281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:53.278935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.319042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3f51079a-bd9f-416e-84f0-d36855170f61 None None] MFA Rules not processed for user `0df60784a8b3468ca19ec2cee1d929e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:53.320518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3f51079a-bd9f-416e-84f0-d36855170f61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.421422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:53.445598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:53.469431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 851/1693] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:53 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 244 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:53.479502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.480065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.480507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.491062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 843/1694] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:53.507655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:53.510581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.515145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] MFA Rules not processed for user `0df60784a8b3468ca19ec2cee1d929e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:53.518922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.543142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 852/1695] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:53 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:53.554794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:53.556447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.685351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20473d7f-5c41-4c73-a2bc-762b113b6acc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:53.701021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93767eb7-5a2f-48f0-b036-314af1e42db7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:53.701706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93767eb7-5a2f-48f0-b036-314af1e42db7 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.702055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93767eb7-5a2f-48f0-b036-314af1e42db7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.702312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93767eb7-5a2f-48f0-b036-314af1e42db7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.726607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93767eb7-5a2f-48f0-b036-314af1e42db7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:53.731511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93767eb7-5a2f-48f0-b036-314af1e42db7 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:53.760084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 844/1696] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:53.762195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943d3b16-b423-4167-ab5f-f3e3355170f0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.762490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943d3b16-b423-4167-ab5f-f3e3355170f0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.762762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-943d3b16-b423-4167-ab5f-f3e3355170f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.829682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-943d3b16-b423-4167-ab5f-f3e3355170f0 None None] MFA Rules not processed for user `d0ed4854e0d449e58d09487fe8eedc2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:53.831816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-943d3b16-b423-4167-ab5f-f3e3355170f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:53.844038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b1efdf-c5b1-4073-891c-dc649f0e7039 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:53.863898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:53.864560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:53.864875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:53.871018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:53.907144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:53.908882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:53.961396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 853/1697] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:53 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:12:54.038487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 845/1698] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:53 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:54.048384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.049861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.092680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c50aed3-d995-4805-9879-545fa6260962 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.131554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.133765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.135057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.138128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.222080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0dd3a0-fa34-44bc-8fff-786359076cda None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.223632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0dd3a0-fa34-44bc-8fff-786359076cda None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.224350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a0dd3a0-fa34-44bc-8fff-786359076cda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.238218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.254028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:54.284743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6a0dd3a0-fa34-44bc-8fff-786359076cda None None] MFA Rules not processed for user `480c3f60b5e14b5f93e5dc921694aee4`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:54.291196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6a0dd3a0-fa34-44bc-8fff-786359076cda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.305111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 854/1699] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:12:54 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 257 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:12:54.481940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 846/1700] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:54 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:12:54.682525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.684873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:54.707500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40ea2e6-e7d9-4451-971c-170f6373819d service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:54.714955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:54.716500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:54.716925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:54.717358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] PATH_INFO: `/v3/projects/eed46241faa14306b5a4661fbd169c25` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:54.718989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=eed46241faa14306b5a4661fbd169c25)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:54.720766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: WARNING py.warnings [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 29 02:12:54.720766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: warnings.warn(msg) Sep 29 02:12:54.720766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: Sep 29 02:12:54.722488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:54.812013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eed46241faa14306b5a4661fbd169c25', 'request_id': 'req-24ec88c8-232c-4b54-873d-b2dd7cef17b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.825721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eed46241faa14306b5a4661fbd169c25', 'request_id': 'req-24ec88c8-232c-4b54-873d-b2dd7cef17b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.854655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eed46241faa14306b5a4661fbd169c25', 'request_id': 'req-24ec88c8-232c-4b54-873d-b2dd7cef17b6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.892022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] The token cache is being invalidate because project eed46241faa14306b5a4661fbd169c25 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:54.892567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.894361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-24ec88c8-232c-4b54-873d-b2dd7cef17b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:54.896469 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 855/1701] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:54 2022] DELETE /identity/v3/projects/eed46241faa14306b5a4661fbd169c25 => generated 0 bytes in 214 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:54.901383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0dd3a0-fa34-44bc-8fff-786359076cda None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:54.923773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a0dd3a0-fa34-44bc-8fff-786359076cda None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:55.061363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:55.119694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:55.120683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.121285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.121762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] PATH_INFO: `/v3/users/2179461e07bd4ef6a120702ba6d07c8d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:55.123369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] RBAC: Authorizing `identity:delete_user(user_id=2179461e07bd4ef6a120702ba6d07c8d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:55.125578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:55.245092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2179461e07bd4ef6a120702ba6d07c8d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.258255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2179461e07bd4ef6a120702ba6d07c8d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.260121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2179461e07bd4ef6a120702ba6d07c8d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.262227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2179461e07bd4ef6a120702ba6d07c8d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.294654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-20f95334-05b8-4684-a0a8-f23d2f16d76c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2179461e07bd4ef6a120702ba6d07c8d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:55.406413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 847/1702] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:12:54 2022] DELETE /identity/v3/users/2179461e07bd4ef6a120702ba6d07c8d => generated 0 bytes in 506 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:55.992151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:55.993148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:55.993762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.094377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:56.111241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:56.117613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:56.369745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 856/1703] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:12:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 378 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:56.600316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.601317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.601841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:56.624084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:56.627313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:56.698620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] MFA Rules not processed for user `0df60784a8b3468ca19ec2cee1d929e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:56.707134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:56.856721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 848/1704] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:56 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 255 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:56.970233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5aed6e7d-7ce7-4d75-8266-1989f7322b74 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:56.984543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3798a4a2-6247-45b5-8056-6a6cd66bd4cc service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:56.985709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3798a4a2-6247-45b5-8056-6a6cd66bd4cc service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:56.987089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3798a4a2-6247-45b5-8056-6a6cd66bd4cc service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:56.987089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3798a4a2-6247-45b5-8056-6a6cd66bd4cc service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.028580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3798a4a2-6247-45b5-8056-6a6cd66bd4cc service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.030972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3798a4a2-6247-45b5-8056-6a6cd66bd4cc service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.091221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 857/1705] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:12:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 467 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:12:57.253708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:57.255226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:57.596306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ad2bba-70c9-4caf-a930-06432a26e5b0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:57.620990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:57.623480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.623943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.624415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] PATH_INFO: `/v3/projects/9d2fdb2758d241c38ce51f2651c6c937` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:57.630998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9d2fdb2758d241c38ce51f2651c6c937)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:57.638073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:57.651417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:57.840707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9d2fdb2758d241c38ce51f2651c6c937', 'request_id': 'req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:57.869382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9d2fdb2758d241c38ce51f2651c6c937', 'request_id': 'req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:57.879215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9d2fdb2758d241c38ce51f2651c6c937', 'request_id': 'req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:57.921308 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] The token cache is being invalidate because project 9d2fdb2758d241c38ce51f2651c6c937 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:12:57.923372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:57.926644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9c33cde6-2c28-4c0e-b00e-1c94a4aee163 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:12:57.928989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 849/1706] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:12:57 2022] DELETE /identity/v3/projects/9d2fdb2758d241c38ce51f2651c6c937 => generated 0 bytes in 676 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:12:57.937852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8240e-2d85-4a21-bb94-ec1ef7f003e7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:57.939700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8240e-2d85-4a21-bb94-ec1ef7f003e7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:57.940734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36f8240e-2d85-4a21-bb94-ec1ef7f003e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.051764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-36f8240e-2d85-4a21-bb94-ec1ef7f003e7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:58.058864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-36f8240e-2d85-4a21-bb94-ec1ef7f003e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:58.401779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 858/1707] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 468 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:58.409895 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86d5f8-3981-4183-842d-7b76d230aaf5 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.410432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86d5f8-3981-4183-842d-7b76d230aaf5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.410845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb86d5f8-3981-4183-842d-7b76d230aaf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.524304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-fb86d5f8-3981-4183-842d-7b76d230aaf5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:58.525922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-fb86d5f8-3981-4183-842d-7b76d230aaf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:58.578466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 850/1708] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:58.579718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8240e-2d85-4a21-bb94-ec1ef7f003e7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.600198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36f8240e-2d85-4a21-bb94-ec1ef7f003e7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.662437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.663111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.663386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.663659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.664323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:12:58.664624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.669150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.717856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:12:58.750782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86d5f8-3981-4183-842d-7b76d230aaf5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.752708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 859/1709] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:12:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:12:58.810726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb86d5f8-3981-4183-842d-7b76d230aaf5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:58.838149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:58.839393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:58.839796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:58.840125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:58.840901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:58.851885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:58.912086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 851/1710] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/projects => generated 401 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:58.916506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:58.936926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4f3254-b297-47e1-909d-862aa4001fe8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.129377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.130372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.130612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.131144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.132512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.137560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.216851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 860/1711] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:12:58 2022] POST /identity/v3/users => generated 301 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:12:59.220752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:59.252736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d2bd70f-a70f-447c-afba-15279dc1363b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.276948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.278045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.278726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.279079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.280025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.283985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.342868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 852/1712] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:59 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:59.344373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:59.358977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f69ced-c512-4e28-b16d-50f4a5df48ad admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.378383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2403e1dc-6848-4cf7-8491-163ffd84747e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.379236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2403e1dc-6848-4cf7-8491-163ffd84747e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.379664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2403e1dc-6848-4cf7-8491-163ffd84747e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.380086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2403e1dc-6848-4cf7-8491-163ffd84747e admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.381052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2403e1dc-6848-4cf7-8491-163ffd84747e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.383013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2403e1dc-6848-4cf7-8491-163ffd84747e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.407063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 861/1713] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:12:59 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:12:59.411054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:12:59.432928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eafbd8f4-1096-44a7-bbc5-f67c48b3515f admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:12:59.451031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:12:59.451892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.452326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.452742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] PATH_INFO: `/v3/projects/9abd8cb7ebaa49419984c0d57ac093d1/users/9a240e2c8287431db8ee7ca7c94eb324/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.509121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9abd8cb7ebaa49419984c0d57ac093d1, user_id=9a240e2c8287431db8ee7ca7c94eb324, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:12:59.512896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:12:59.600571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 853/1714] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:12:59 2022] PUT /identity/v3/projects/9abd8cb7ebaa49419984c0d57ac093d1/users/9a240e2c8287431db8ee7ca7c94eb324/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:12:59.610806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7534eb-bc72-4e63-a065-a7e21e4a09bd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:12:59.611322 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7534eb-bc72-4e63-a065-a7e21e4a09bd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:12:59.611776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7534eb-bc72-4e63-a065-a7e21e4a09bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:12:59.702839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9d7534eb-bc72-4e63-a065-a7e21e4a09bd None None] MFA Rules not processed for user `9a240e2c8287431db8ee7ca7c94eb324`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:12:59.704377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9d7534eb-bc72-4e63-a065-a7e21e4a09bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:12:59.862782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 862/1715] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:12:59 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 253 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:12:59.867151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:00.082201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5eeb11fd-f150-4f5f-a561-3f1e4c656457 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:00.119001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:00.119990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:00.120519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:00.120999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:00.121869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:00.124123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:00.239120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 854/1716] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:12:59 2022] POST /identity/v3/projects => generated 401 bytes in 373 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:00.242801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7534eb-bc72-4e63-a065-a7e21e4a09bd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:00.272384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7534eb-bc72-4e63-a065-a7e21e4a09bd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:00.441085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:00.441893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:00.442101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:00.442379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:00.443271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:00.446099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:00.477197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:00.499842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 863/1717] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:00 2022] POST /identity/v3/users => generated 307 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:00.502380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:00.521450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a645c3-2900-472a-bfe3-d8ba5437ec80 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:00.556169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:00.557876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:00.558189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:00.558720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:00.561992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:00.564730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:00.601440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:00.609367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 855/1718] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:00.626150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbe68c83-7059-47c0-a817-2d5432f66ff0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:00.634887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:00.635584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:00.635864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:00.636136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] PATH_INFO: `/v3/projects/2ab2f3b4d4334e9ea98f75a4dfd9dbc5/users/ff4f0763dc004df28de3da60f3e844bf/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:00.742219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ab2f3b4d4334e9ea98f75a4dfd9dbc5, user_id=ff4f0763dc004df28de3da60f3e844bf, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:00.748860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:00.783276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 864/1719] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:00 2022] PUT /identity/v3/projects/2ab2f3b4d4334e9ea98f75a4dfd9dbc5/users/ff4f0763dc004df28de3da60f3e844bf/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:00.787562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:00.796195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:00.813425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7a9e34-636b-41aa-ac98-e72c2ff92a80 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:01.220000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-146c6a0d-ce69-4013-bcdb-3a6b39acc510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:01.220695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c6a0d-ce69-4013-bcdb-3a6b39acc510 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:01.221024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c6a0d-ce69-4013-bcdb-3a6b39acc510 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:01.221290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-146c6a0d-ce69-4013-bcdb-3a6b39acc510 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:01.222166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146c6a0d-ce69-4013-bcdb-3a6b39acc510 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:01.223857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-146c6a0d-ce69-4013-bcdb-3a6b39acc510 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:01.268154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 856/1720] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:00 2022] GET /identity/v3/roles => generated 1420 bytes in 481 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:01.272771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:01.308589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc06c7f-9835-4d0c-ae09-916bbc3143ff admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:01.328442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:01.329307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:01.331392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:01.333636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] PATH_INFO: `/v3/projects/2ab2f3b4d4334e9ea98f75a4dfd9dbc5/users/ff4f0763dc004df28de3da60f3e844bf/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:01.355861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2ab2f3b4d4334e9ea98f75a4dfd9dbc5, user_id=ff4f0763dc004df28de3da60f3e844bf, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:01.377888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:01.456164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 865/1721] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:01 2022] PUT /identity/v3/projects/2ab2f3b4d4334e9ea98f75a4dfd9dbc5/users/ff4f0763dc004df28de3da60f3e844bf/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:01.468094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65abc9bf-e620-4b6f-a1c9-6bd92d07df74 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:01.468670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65abc9bf-e620-4b6f-a1c9-6bd92d07df74 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:01.469697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65abc9bf-e620-4b6f-a1c9-6bd92d07df74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:01.593588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-65abc9bf-e620-4b6f-a1c9-6bd92d07df74 None None] MFA Rules not processed for user `ff4f0763dc004df28de3da60f3e844bf`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:01.596027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-65abc9bf-e620-4b6f-a1c9-6bd92d07df74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:01.930676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 857/1722] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:01 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 464 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:01.943449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:01.945353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.181511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64dba856-9131-4b29-a3ab-9ada0c805882 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:02.205651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:02.212731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.213205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.213534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.243358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:02.244937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:02.308383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 866/1723] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:01 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:02.342762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.343701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.344116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.407875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] MFA Rules not processed for user `cc2f041fe7194a3da199517b4cdbd265`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:02.415374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.746103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 858/1724] 10.222.0.55 () {58 vars in 1015 bytes} [Thu Sep 29 02:13:02 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 405 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:02.835080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:02.837458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.861897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b34f0f7b-ed50-4693-a25a-a62a7ce6a438 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:02.867084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4252739-2eab-4e10-9446-7d1c9d2bb2d7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:02.867951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4252739-2eab-4e10-9446-7d1c9d2bb2d7 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.868432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4252739-2eab-4e10-9446-7d1c9d2bb2d7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.869505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4252739-2eab-4e10-9446-7d1c9d2bb2d7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.879171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4252739-2eab-4e10-9446-7d1c9d2bb2d7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:02.881246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4252739-2eab-4e10-9446-7d1c9d2bb2d7 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:02.894354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 867/1725] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:02 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 60 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:02.897717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:02.899551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:02.920556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.921076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.921528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.945541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6293a6da-87a6-41b6-875c-43b8be0608e4 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:02.956157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:02.959047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:02.961037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:02.961383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:02.998532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] MFA Rules not processed for user `d0ed4854e0d449e58d09487fe8eedc2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:03.008091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.083002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:03.139519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:03.145950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:03.182313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 859/1726] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:02 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:03.184630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:03.190234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:03.235929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-784e8f5e-feed-43df-b742-a5b47c047a94 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:03.242318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf0f1c8-16ea-49df-b34f-66a5dab357e7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:03.243471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf0f1c8-16ea-49df-b34f-66a5dab357e7 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:03.243961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf0f1c8-16ea-49df-b34f-66a5dab357e7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:03.244411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf0f1c8-16ea-49df-b34f-66a5dab357e7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:03.249877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 868/1727] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:02 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:03.283944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf0f1c8-16ea-49df-b34f-66a5dab357e7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:03.290578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf0f1c8-16ea-49df-b34f-66a5dab357e7 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:03.322946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 860/1728] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:04.844256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:04.845777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.035827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:05.073904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af6a23a1-6e12-45c8-9eb1-06a3d55bc5c4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:05.090933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:05.091645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.091925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.092199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] PATH_INFO: `/v3/users/e44e3c8ac0244d598fda750643355e1e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.097384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e44e3c8ac0244d598fda750643355e1e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:05.104379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:05.247414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e44e3c8ac0244d598fda750643355e1e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.279426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e44e3c8ac0244d598fda750643355e1e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.294048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e44e3c8ac0244d598fda750643355e1e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.296770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e44e3c8ac0244d598fda750643355e1e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.319698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e44e3c8ac0244d598fda750643355e1e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:05.369183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 869/1729] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:04 2022] DELETE /identity/v3/users/e44e3c8ac0244d598fda750643355e1e => generated 0 bytes in 530 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:05.375563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa810cb-23df-4311-ba31-876a4f20a7ea None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:05.376054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa810cb-23df-4311-ba31-876a4f20a7ea None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:05.377054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fa810cb-23df-4311-ba31-876a4f20a7ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:05.448278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6fa810cb-23df-4311-ba31-876a4f20a7ea None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:05.453303 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6fa810cb-23df-4311-ba31-876a4f20a7ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:05.748132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 861/1730] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:05.814678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:05.816739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:06.066699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d0349aa-5776-41df-91bf-44f8d733c3c5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:06.094819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-236d9a40-2b76-45fe-8844-a8009084e05a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:06.095537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d9a40-2b76-45fe-8844-a8009084e05a service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:06.098125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d9a40-2b76-45fe-8844-a8009084e05a service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:06.098494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236d9a40-2b76-45fe-8844-a8009084e05a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:06.131664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236d9a40-2b76-45fe-8844-a8009084e05a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:06.134911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-236d9a40-2b76-45fe-8844-a8009084e05a service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:06.152631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 870/1731] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:08.631491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97232d3f-5514-492a-8368-67f022bc4e3f None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.632038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97232d3f-5514-492a-8368-67f022bc4e3f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.632038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97232d3f-5514-492a-8368-67f022bc4e3f None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.633647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 862/1732] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:13:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:13:08.722792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.723424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.723599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.765041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:08.766657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:08.811595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ded4174-aa15-4a65-9abd-71ed9c615308 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.812362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ded4174-aa15-4a65-9abd-71ed9c615308 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.812822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ded4174-aa15-4a65-9abd-71ed9c615308 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.814687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 863/1733] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:13:08 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 29 02:13:08.842438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef25427b-b701-4761-b555-98a81e3c0aec None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:08.843264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef25427b-b701-4761-b555-98a81e3c0aec None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:08.844061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef25427b-b701-4761-b555-98a81e3c0aec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:08.891339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ef25427b-b701-4761-b555-98a81e3c0aec None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:08.893699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ef25427b-b701-4761-b555-98a81e3c0aec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:08.961390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 871/1734] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 241 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:09.083157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 864/1735] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:13:08 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:09.575580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:09.577666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:09.619759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63b6711-8a7f-46f7-9862-ec0e3800981d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:09.630297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:09.631195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.631615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.632491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] PATH_INFO: `/v3/users/9a240e2c8287431db8ee7ca7c94eb324` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.635863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a240e2c8287431db8ee7ca7c94eb324)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:09.640161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:09.663058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:09.769153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a240e2c8287431db8ee7ca7c94eb324'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.784124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a240e2c8287431db8ee7ca7c94eb324'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.785624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a240e2c8287431db8ee7ca7c94eb324'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.786968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a240e2c8287431db8ee7ca7c94eb324'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.807806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a240e2c8287431db8ee7ca7c94eb324'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:09.841341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 872/1736] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:09 2022] DELETE /identity/v3/users/9a240e2c8287431db8ee7ca7c94eb324 => generated 0 bytes in 267 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:09.845907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:09.846566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:09.846803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:09.902729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:09.918514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.079663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:10.151489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 865/1737] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:10.214660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.216976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.320173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.322339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.326278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25d14909-5249-4365-b228-1eb02d3c2dd9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.330829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.331508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.331944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.332218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.352801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.354502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badb383f-5064-4c24-a9fb-83f906d3da1f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.358281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.374654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 873/1738] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:10.376115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.376907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.377672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.377672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] PATH_INFO: `/v3/projects/6c8694988d8e4d72a26790d23a332f2c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.382953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c8694988d8e4d72a26790d23a332f2c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.385281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.483565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c8694988d8e4d72a26790d23a332f2c', 'request_id': 'req-3fcbd514-a1a7-4f54-8dfc-b179576858b5'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.495820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c8694988d8e4d72a26790d23a332f2c', 'request_id': 'req-3fcbd514-a1a7-4f54-8dfc-b179576858b5'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.506556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c8694988d8e4d72a26790d23a332f2c', 'request_id': 'req-3fcbd514-a1a7-4f54-8dfc-b179576858b5'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.528268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] The token cache is being invalidate because project 6c8694988d8e4d72a26790d23a332f2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:10.528737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.529567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.534228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 866/1739] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:10 2022] DELETE /identity/v3/projects/6c8694988d8e4d72a26790d23a332f2c => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:10.535556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.553990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52927561-e78a-4c27-806d-89b8a3565998 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:10.630582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:10.631236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:10.631513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:10.631960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] PATH_INFO: `/v3/users/5d28e1984ad44d53af3df1b99c4678ad` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:10.641705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] RBAC: Authorizing `identity:delete_user(user_id=5d28e1984ad44d53af3df1b99c4678ad)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:10.644639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:10.704935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:10.758339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5d28e1984ad44d53af3df1b99c4678ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.777895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5d28e1984ad44d53af3df1b99c4678ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.780508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5d28e1984ad44d53af3df1b99c4678ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.781165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5d28e1984ad44d53af3df1b99c4678ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.820985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:10.822661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:10.836970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5d28e1984ad44d53af3df1b99c4678ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:10.869548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 874/1740] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:10 2022] DELETE /identity/v3/users/5d28e1984ad44d53af3df1b99c4678ad => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:10.945884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fcbd514-a1a7-4f54-8dfc-b179576858b5 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.067674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.068643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.069534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.070053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.073167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.103879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.105841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.129272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:11.143276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.145055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.169896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f12585-2272-4744-a65f-e27a866acf0b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.177632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 867/1741] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:10 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 357 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:11.180118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.181346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.182041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.182674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.184793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.209342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.211139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.249856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 875/1742] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:11 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:11.295299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.296943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.480362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03088d1d-ea20-47c6-bdb8-4140db45dc5d service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.487434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.489319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.489753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.490266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] PATH_INFO: `/v3/projects/9abd8cb7ebaa49419984c0d57ac093d1` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.492723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] RBAC: Authorizing `identity:delete_project(project_id=9abd8cb7ebaa49419984c0d57ac093d1)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.498119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.600724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9abd8cb7ebaa49419984c0d57ac093d1', 'request_id': 'req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.612933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9abd8cb7ebaa49419984c0d57ac093d1', 'request_id': 'req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.617986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.619090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.630671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9abd8cb7ebaa49419984c0d57ac093d1', 'request_id': 'req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.652208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] The token cache is being invalidate because project 9abd8cb7ebaa49419984c0d57ac093d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:11.652481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.653280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.659250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b65de17-b704-4e9f-81f1-4511e932af7f service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.660081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 868/1743] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:11 2022] DELETE /identity/v3/projects/9abd8cb7ebaa49419984c0d57ac093d1 => generated 0 bytes in 361 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:11.661142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:11.662856 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:11.779884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.780548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.780979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.781377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] PATH_INFO: `/v3/projects/8e763b80ea264b42a88d1507fdc35830` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.783470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] RBAC: Authorizing `identity:delete_project(project_id=8e763b80ea264b42a88d1507fdc35830)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.786726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.816962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27d55dc7-a3b7-4499-bcb6-9647cb4e635b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:11.825382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:11.826132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.827260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.827557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] PATH_INFO: `/v3/users/ff4f0763dc004df28de3da60f3e844bf` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.829564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff4f0763dc004df28de3da60f3e844bf)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:11.832183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:11.892853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:11.898608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8e763b80ea264b42a88d1507fdc35830', 'request_id': 'req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.922677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8e763b80ea264b42a88d1507fdc35830', 'request_id': 'req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.932691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8e763b80ea264b42a88d1507fdc35830', 'request_id': 'req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.939651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff4f0763dc004df28de3da60f3e844bf'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.954032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] The token cache is being invalidate because project 8e763b80ea264b42a88d1507fdc35830 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:11.954335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff4f0763dc004df28de3da60f3e844bf'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.954792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.956712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff4f0763dc004df28de3da60f3e844bf'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.957128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bc538714-3e6a-4f66-abd0-a8b9e51e43cb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.957676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff4f0763dc004df28de3da60f3e844bf'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:11.964709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 876/1744] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:11 2022] DELETE /identity/v3/projects/8e763b80ea264b42a88d1507fdc35830 => generated 0 bytes in 342 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:11.969122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79aea03-4ecc-4fb7-99e4-b3ba8e1b9f96 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:11.971409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79aea03-4ecc-4fb7-99e4-b3ba8e1b9f96 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:11.971697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79aea03-4ecc-4fb7-99e4-b3ba8e1b9f96 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:11.990189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f552425e-bf7b-4e5a-95c2-d9528820ab22 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff4f0763dc004df28de3da60f3e844bf'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:12.011468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c79aea03-4ecc-4fb7-99e4-b3ba8e1b9f96 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:12.013235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c79aea03-4ecc-4fb7-99e4-b3ba8e1b9f96 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:12.044382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:12.053334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 869/1745] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:11 2022] DELETE /identity/v3/users/ff4f0763dc004df28de3da60f3e844bf => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:12.182036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 877/1746] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:12.183710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16debfc9-3ba8-42b5-a74d-bd66644be289 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:12.184001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16debfc9-3ba8-42b5-a74d-bd66644be289 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:12.184330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16debfc9-3ba8-42b5-a74d-bd66644be289 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:12.246398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-16debfc9-3ba8-42b5-a74d-bd66644be289 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:12.249623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-16debfc9-3ba8-42b5-a74d-bd66644be289 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:12.355816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79aea03-4ecc-4fb7-99e4-b3ba8e1b9f96 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:12.363263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 870/1747] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:12.368050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79aea03-4ecc-4fb7-99e4-b3ba8e1b9f96 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:12.375290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:12.375979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:12.376249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:12.376675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:12.377472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:12.377851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:12.379750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:12.404435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16debfc9-3ba8-42b5-a74d-bd66644be289 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:12.411551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 878/1748] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:12.431792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16debfc9-3ba8-42b5-a74d-bd66644be289 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:12.447720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:12.449351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:12.449637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:12.449910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:12.455047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:12.457168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:12.543739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 871/1749] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:12 2022] POST /identity/v3/projects => generated 365 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:12.548443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:12.565537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c097af2-0337-4fc7-9ca0-a8fac877fbae admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:12.801116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:12.805306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:12.806312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:12.809225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:12.811932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:12.815793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:12.875388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:12.878974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 879/1750] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:12 2022] POST /identity/v3/users => generated 283 bytes in 330 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:12.883724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:12.883839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:12.908521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffedccd-001a-465a-94d7-ee35f75305b6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:12.918311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:12.919542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:12.919952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:12.920679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] PATH_INFO: `/v3/projects/2ab2f3b4d4334e9ea98f75a4dfd9dbc5` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:12.922821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2ab2f3b4d4334e9ea98f75a4dfd9dbc5)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:12.924937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:12.950814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa76198-1bd6-4c09-a897-d9688426280d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:12.958171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:12.959001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:12.959433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:12.959858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:12.960719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:12.962732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:12.978824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 880/1751] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:12 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:12.983680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.023502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-149ba840-7e7f-412d-826f-c36827b17fd7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.052323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2ab2f3b4d4334e9ea98f75a4dfd9dbc5', 'request_id': 'req-0e27af9d-6a12-4c87-9aab-1a958206e442'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:13.070349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2ab2f3b4d4334e9ea98f75a4dfd9dbc5', 'request_id': 'req-0e27af9d-6a12-4c87-9aab-1a958206e442'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:13.081027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2ab2f3b4d4334e9ea98f75a4dfd9dbc5', 'request_id': 'req-0e27af9d-6a12-4c87-9aab-1a958206e442'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:13.096295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] The token cache is being invalidate because project 2ab2f3b4d4334e9ea98f75a4dfd9dbc5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:13.096588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:13.099897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0e27af9d-6a12-4c87-9aab-1a958206e442 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:13.101732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 872/1752] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:12 2022] DELETE /identity/v3/projects/2ab2f3b4d4334e9ea98f75a4dfd9dbc5 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:13.109230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24103925-e789-4241-8303-35232c054815 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.109576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24103925-e789-4241-8303-35232c054815 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.109819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24103925-e789-4241-8303-35232c054815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.150596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-24103925-e789-4241-8303-35232c054815 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:13.152346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-24103925-e789-4241-8303-35232c054815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:13.158111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:13.169413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.170357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.170940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.171218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.171871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.173689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.195954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 881/1753] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:12 2022] GET /identity/v3/roles => generated 1420 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:13.197101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.200094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:13.217929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa0eabe-4da8-4961-844a-30b777da44b2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.223375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f087f64a-5b44-46db-8b35-2191eb54b865 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.224588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f087f64a-5b44-46db-8b35-2191eb54b865 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.224973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f087f64a-5b44-46db-8b35-2191eb54b865 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.225487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f087f64a-5b44-46db-8b35-2191eb54b865 admin admin] PATH_INFO: `/v3/projects/fdb8dc40956f41938b60febf378f2690/users/948461ec333c4dc49d2fd11f5759ea42/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.243951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f087f64a-5b44-46db-8b35-2191eb54b865 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fdb8dc40956f41938b60febf378f2690, user_id=948461ec333c4dc49d2fd11f5759ea42, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.247813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f087f64a-5b44-46db-8b35-2191eb54b865 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.251640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:13.266291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 873/1754] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:13.268183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c13f35-e8f5-4fee-9992-5652205759ab None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.268298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c13f35-e8f5-4fee-9992-5652205759ab None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.268565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c13f35-e8f5-4fee-9992-5652205759ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.275131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 882/1755] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:13 2022] PUT /identity/v3/projects/fdb8dc40956f41938b60febf378f2690/users/948461ec333c4dc49d2fd11f5759ea42/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 78 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:13.294271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2271195-9336-4cae-93e4-8b7dfd2b35e0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.294635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2271195-9336-4cae-93e4-8b7dfd2b35e0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.295467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2271195-9336-4cae-93e4-8b7dfd2b35e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.307200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-01c13f35-e8f5-4fee-9992-5652205759ab None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:13.309400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-01c13f35-e8f5-4fee-9992-5652205759ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:13.332450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b2271195-9336-4cae-93e4-8b7dfd2b35e0 None None] MFA Rules not processed for user `948461ec333c4dc49d2fd11f5759ea42`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:13.333847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b2271195-9336-4cae-93e4-8b7dfd2b35e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:13.428534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 874/1756] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:13.433071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c13f35-e8f5-4fee-9992-5652205759ab None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.451565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01c13f35-e8f5-4fee-9992-5652205759ab None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.457855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.458760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.458760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.459049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.459815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:13.460134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.462254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.478026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 875/1757] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:13.483248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.493830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf3c27c6-8393-4a46-8ba9-3d9a2e6d8396 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.499703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.500330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.500604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.500870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.501510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.506462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 883/1758] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:13 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:13.506875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.508521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2271195-9336-4cae-93e4-8b7dfd2b35e0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.560874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 876/1759] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:13 2022] POST /identity/v3/projects => generated 383 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:13.564867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.569327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2271195-9336-4cae-93e4-8b7dfd2b35e0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.701834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.702551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.702837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.703100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.703749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.705714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.756424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 884/1760] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:13 2022] POST /identity/v3/projects => generated 363 bytes in 248 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:13.759514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ec9f-28ab-434e-81f9-b9b2b61be8c6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.763071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.781848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:13.834338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.834990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.835266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.835530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.836208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.838192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.847946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e61836-8deb-4e2b-8be5-1242896e19a7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.852754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.853358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.853628 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.853955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.854658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.856570 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.864699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 877/1761] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:13 2022] POST /identity/v3/users => generated 292 bytes in 300 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:13.868754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.878986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:13.880768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd22d0e-4f6c-4d73-a58f-9cc3265362a6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.886539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.887202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.887612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.887723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 885/1762] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:13 2022] POST /identity/v3/users => generated 288 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:13.888019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.888766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.888919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.890692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.898806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2321b186-9e1e-461c-85de-1442cbfe57d8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.904612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 878/1763] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:13 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:13.905214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.906366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.908386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.908872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.909322 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.910191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.912221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.917743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d4c774-061e-4b0e-be81-73b211e80b27 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.922876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.923476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.923751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.924070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.926043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 886/1764] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:13 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:13.926181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.926261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.929071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.937670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 879/1765] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:13 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:13.940080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860f7612-c5d5-47ba-8701-ace2e733a9e2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.941719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:13.944782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.945448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.945720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.945991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] PATH_INFO: `/v3/projects/e65c06949e1c4d02ada8f42dc10c2ed3/users/990eae56faa54e7a9fcdcff8f4906747/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.955110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f759784b-e393-47a5-b1ea-310cd3d371ad admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:13.960970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aea2ad3-eae0-4c07-b2a6-60d768b3e88d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:13.961813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aea2ad3-eae0-4c07-b2a6-60d768b3e88d admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:13.962233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aea2ad3-eae0-4c07-b2a6-60d768b3e88d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:13.962584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aea2ad3-eae0-4c07-b2a6-60d768b3e88d admin admin] PATH_INFO: `/v3/projects/669a4d6f48a44bc59f67dc8e27bf656e/users/1a612fd2c2604f07b490b7cf86133700/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:13.964236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e65c06949e1c4d02ada8f42dc10c2ed3, user_id=990eae56faa54e7a9fcdcff8f4906747, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.967625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.981455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aea2ad3-eae0-4c07-b2a6-60d768b3e88d admin admin] RBAC: Authorizing `identity:create_grant(project_id=669a4d6f48a44bc59f67dc8e27bf656e, user_id=1a612fd2c2604f07b490b7cf86133700, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:13.983917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 887/1766] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:13 2022] PUT /identity/v3/projects/e65c06949e1c4d02ada8f42dc10c2ed3/users/990eae56faa54e7a9fcdcff8f4906747/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:13.986903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aea2ad3-eae0-4c07-b2a6-60d768b3e88d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:13.987715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.010390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 880/1767] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:13 2022] PUT /identity/v3/projects/669a4d6f48a44bc59f67dc8e27bf656e/users/1a612fd2c2604f07b490b7cf86133700/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:14.021240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e91e4c-860d-4eff-ab0c-d45e063d8240 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.021532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e91e4c-860d-4eff-ab0c-d45e063d8240 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.021781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e91e4c-860d-4eff-ab0c-d45e063d8240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.050025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c6e91e4c-860d-4eff-ab0c-d45e063d8240 None None] MFA Rules not processed for user `1a612fd2c2604f07b490b7cf86133700`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:14.050382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136f007a-03e8-4fc5-931b-395e254fe91d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.051544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c6e91e4c-860d-4eff-ab0c-d45e063d8240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:14.106680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.107297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.107571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.107841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.108511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.110169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.116709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 881/1768] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:14 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:14.121155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 888/1769] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:13 2022] GET /identity/v3/roles => generated 1420 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:14.121627 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e91e4c-860d-4eff-ab0c-d45e063d8240 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.124922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.132860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e91e4c-860d-4eff-ab0c-d45e063d8240 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.134401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dad955ca-4426-4d7f-9cb0-8aad94cbe709 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.137921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.138560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.138839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.139108 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.139575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8407d2d7-c366-4717-9559-2256e4d85a10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.139743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.140207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407d2d7-c366-4717-9559-2256e4d85a10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.140483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407d2d7-c366-4717-9559-2256e4d85a10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.140756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8407d2d7-c366-4717-9559-2256e4d85a10 admin admin] PATH_INFO: `/v3/projects/e65c06949e1c4d02ada8f42dc10c2ed3/users/990eae56faa54e7a9fcdcff8f4906747/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.141663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.142699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407d2d7-c366-4717-9559-2256e4d85a10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e65c06949e1c4d02ada8f42dc10c2ed3, user_id=990eae56faa54e7a9fcdcff8f4906747, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.145293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8407d2d7-c366-4717-9559-2256e4d85a10 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.157659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 889/1770] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:14 2022] PUT /identity/v3/projects/e65c06949e1c4d02ada8f42dc10c2ed3/users/990eae56faa54e7a9fcdcff8f4906747/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:14.169676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.169971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.170293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.172456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 882/1771] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:14 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:14.176184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.201773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] MFA Rules not processed for user `990eae56faa54e7a9fcdcff8f4906747`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:14.203038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:14.214140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:14.247657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3a4ca3b-3bd8-4fdf-aa7e-82f57fb388ca admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.254968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.255221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:14.256160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.256594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.256956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.260276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.262811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.290501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 883/1772] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:14 2022] POST /identity/v3/users => generated 298 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:14.293534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 890/1773] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:14 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:14.300841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.306553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.308246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:14.311954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbf2c3f-191d-4412-a808-3f10495d349e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.316709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.317318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.317589 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.317877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.318534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.320128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.330587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 884/1774] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:14.334924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.345314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9824737f-78c5-4fbe-8d8e-3541f9665b1b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.350038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.350729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.350999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.351264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] PATH_INFO: `/v3/projects/14a661beab454b59a3de09569667accb/users/c452fa9bc6c84e398660c5e4c6c226de/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.368509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14a661beab454b59a3de09569667accb, user_id=c452fa9bc6c84e398660c5e4c6c226de, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.371506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.375088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f56868-c696-4711-b16e-b6a0cb353043 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.379609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.380225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.380431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.380702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.389850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 885/1775] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:14 2022] PUT /identity/v3/projects/14a661beab454b59a3de09569667accb/users/c452fa9bc6c84e398660c5e4c6c226de/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:14.391014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.396959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.398495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.408585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 891/1776] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:14 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:14.469881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e3ae65-ceac-4765-b963-82f34add3ee0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.475639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.476370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.476694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.477647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.477769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.479506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.494128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 886/1777] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:14 2022] GET /identity/v3/roles => generated 1420 bytes in 103 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:14.495977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.497968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.509323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b17da0-825b-4180-8983-f02002ec46bc service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.517157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc791ad4-735c-479e-a4c2-d61cc4e0ee79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.517877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc791ad4-735c-479e-a4c2-d61cc4e0ee79 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.518719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc791ad4-735c-479e-a4c2-d61cc4e0ee79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.518999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc791ad4-735c-479e-a4c2-d61cc4e0ee79 admin admin] PATH_INFO: `/v3/projects/14a661beab454b59a3de09569667accb/users/c452fa9bc6c84e398660c5e4c6c226de/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.521710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc791ad4-735c-479e-a4c2-d61cc4e0ee79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=14a661beab454b59a3de09569667accb, user_id=c452fa9bc6c84e398660c5e4c6c226de, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.526859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc791ad4-735c-479e-a4c2-d61cc4e0ee79 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.538458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 892/1778] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:14 2022] PUT /identity/v3/projects/14a661beab454b59a3de09569667accb/users/c452fa9bc6c84e398660c5e4c6c226de/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:14.550726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.551554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.551554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.567898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e708c2ee-4c40-4a31-be11-a0c854ccc1d9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.593907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] MFA Rules not processed for user `c452fa9bc6c84e398660c5e4c6c226de`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:14.595199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:14.616914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:14.628963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.629578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.629852 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.630486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.666830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 893/1779] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:14 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:14.678467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.679816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:14.687604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.689184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.692495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2421d85-b6ae-4564-bb7b-9e7f81e33d88 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.696784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.697695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.698688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.699068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.701129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 887/1780] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:14 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:14.709370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.710837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.722212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 894/1781] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:14 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:14.865790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:14.876870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c78a64-e62c-4126-a56c-eef4808daba4 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:14.881937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a35969bd-e6bf-4bab-9bfa-2ed860577b6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:14.882585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35969bd-e6bf-4bab-9bfa-2ed860577b6d service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:14.882862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35969bd-e6bf-4bab-9bfa-2ed860577b6d service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:14.883130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a35969bd-e6bf-4bab-9bfa-2ed860577b6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:14.944839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35969bd-e6bf-4bab-9bfa-2ed860577b6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:14.946440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a35969bd-e6bf-4bab-9bfa-2ed860577b6d service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:14.956273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 888/1782] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:14 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:15.241507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:15.287305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:15.308492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad8164c1-ac1d-44b4-b050-15a90159645a service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:15.314357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:15.315048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:15.315344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:15.315606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] PATH_INFO: `/v3/users/1a612fd2c2604f07b490b7cf86133700` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:15.316755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1a612fd2c2604f07b490b7cf86133700)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:15.318727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:15.370378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1a612fd2c2604f07b490b7cf86133700'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:15.377567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1a612fd2c2604f07b490b7cf86133700'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:15.378272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1a612fd2c2604f07b490b7cf86133700'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:15.378869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1a612fd2c2604f07b490b7cf86133700'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:15.387756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1a612fd2c2604f07b490b7cf86133700'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:15.405911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 895/1783] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:15 2022] DELETE /identity/v3/users/1a612fd2c2604f07b490b7cf86133700 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:15.410951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:15.410951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:15.411425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:15.431547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:15.432968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:15.493360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 889/1784] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:15.503211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:15.504667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:15.567358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6be4cbe-205f-498c-9096-e119cfc691e0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:15.571402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:15.572091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:15.572561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:15.573052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:15.582505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:15.583962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:15.593893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 896/1785] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:16.469359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:16.471598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:16.487017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f45e703-5dd8-4b92-9ffb-ba349e2661eb None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:16.495130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:16.496171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:16.496373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:16.496645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] PATH_INFO: `/v3/projects/669a4d6f48a44bc59f67dc8e27bf656e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:16.498642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] RBAC: Authorizing `identity:delete_project(project_id=669a4d6f48a44bc59f67dc8e27bf656e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:16.501165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:16.551704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:16.573144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '669a4d6f48a44bc59f67dc8e27bf656e', 'request_id': 'req-18dce992-e7fa-42ab-8218-3db97a4fe7ad'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.585500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '669a4d6f48a44bc59f67dc8e27bf656e', 'request_id': 'req-18dce992-e7fa-42ab-8218-3db97a4fe7ad'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.604002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '669a4d6f48a44bc59f67dc8e27bf656e', 'request_id': 'req-18dce992-e7fa-42ab-8218-3db97a4fe7ad'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.617660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] The token cache is being invalidate because project 669a4d6f48a44bc59f67dc8e27bf656e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:16.618067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.618783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.620707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 890/1786] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:16 2022] DELETE /identity/v3/projects/669a4d6f48a44bc59f67dc8e27bf656e => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:16.625100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:16.635838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b839fc7c-fd87-4df9-b3fd-149ae9819521 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:16.707012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:16.708028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:16.708371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:16.708704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] PATH_INFO: `/v3/users/c452fa9bc6c84e398660c5e4c6c226de` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:16.710133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c452fa9bc6c84e398660c5e4c6c226de)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:16.712210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:16.718152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:16.787494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c452fa9bc6c84e398660c5e4c6c226de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.797974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c452fa9bc6c84e398660c5e4c6c226de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.799051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c452fa9bc6c84e398660c5e4c6c226de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.799964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c452fa9bc6c84e398660c5e4c6c226de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.819095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-78a3c7c0-3466-4752-b062-8d767f12d5f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c452fa9bc6c84e398660c5e4c6c226de'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:16.850033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 897/1787] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:16 2022] DELETE /identity/v3/users/c452fa9bc6c84e398660c5e4c6c226de => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:17.268758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:17.270154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:17.301641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dce992-e7fa-42ab-8218-3db97a4fe7ad admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:17.369398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:17.370391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.370684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.370950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] PATH_INFO: `/v3/projects/14a661beab454b59a3de09569667accb` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.372153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] RBAC: Authorizing `identity:delete_project(project_id=14a661beab454b59a3de09569667accb)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:17.374141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:17.420844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '14a661beab454b59a3de09569667accb', 'request_id': 'req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:17.429206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '14a661beab454b59a3de09569667accb', 'request_id': 'req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:17.435778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '14a661beab454b59a3de09569667accb', 'request_id': 'req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:17.445815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] The token cache is being invalidate because project 14a661beab454b59a3de09569667accb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:17.446137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:17.446764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-555f769b-3ac6-43b9-8f7f-f6cdfe4ae353 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:17.448352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 891/1788] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:17 2022] DELETE /identity/v3/projects/14a661beab454b59a3de09569667accb => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:17.453519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902960ba-7857-458c-a2ae-ec95be4dd3c3 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.453809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902960ba-7857-458c-a2ae-ec95be4dd3c3 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.454146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902960ba-7857-458c-a2ae-ec95be4dd3c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.477260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-902960ba-7857-458c-a2ae-ec95be4dd3c3 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:17.478506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-902960ba-7857-458c-a2ae-ec95be4dd3c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:17.560190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 898/1789] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:17.568031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad81fbd-0bc1-4a0d-8771-4bc5d443d62e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.568502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad81fbd-0bc1-4a0d-8771-4bc5d443d62e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.568594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ad81fbd-0bc1-4a0d-8771-4bc5d443d62e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.589698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0ad81fbd-0bc1-4a0d-8771-4bc5d443d62e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:17.591291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0ad81fbd-0bc1-4a0d-8771-4bc5d443d62e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:17.599129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 892/1790] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:17.603548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902960ba-7857-458c-a2ae-ec95be4dd3c3 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:17.613558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902960ba-7857-458c-a2ae-ec95be4dd3c3 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:17.618132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:17.618840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.619144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.619414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.620220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:17.620525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:17.622121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:17.634602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 899/1791] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:17.638890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad81fbd-0bc1-4a0d-8771-4bc5d443d62e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:17.647908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ad81fbd-0bc1-4a0d-8771-4bc5d443d62e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:17.652697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:17.653295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.653565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.653828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.654489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:17.656388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:17.685828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 893/1792] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:17 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:17.690256 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:17.700775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d58e0a-5b73-41ba-a69d-6d5a778a35f8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:17.770668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:17.771314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.771589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.771855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.772522 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:17.774565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:17.794011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:17.802409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 900/1793] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:17 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:17.803882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:17.814746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c95bf76-a659-4d41-aa11-390a0a76fbe0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:17.819725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:17.820331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.820602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.820867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.821503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:17.823261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:17.833575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 894/1794] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:17.837679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:17.848971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67d4c323-f000-48e9-a20f-bb2c678b2562 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:17.854214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-798da073-8bfa-4f2f-8981-2e959f4105e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:17.854804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798da073-8bfa-4f2f-8981-2e959f4105e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.855103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798da073-8bfa-4f2f-8981-2e959f4105e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.855377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-798da073-8bfa-4f2f-8981-2e959f4105e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.856051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798da073-8bfa-4f2f-8981-2e959f4105e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:17.857652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-798da073-8bfa-4f2f-8981-2e959f4105e3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:17.869597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 901/1795] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:17 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:17.873664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:17.883929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d0c7e49-0422-452c-a389-4fc86dd32578 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:17.889429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:17.890367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.890698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.891025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] PATH_INFO: `/v3/projects/6f18edbbd7d14f6aaebe992d45b60abc/users/fe764e185aea4545974fcc05498b6b79/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.898395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:17.914236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6f18edbbd7d14f6aaebe992d45b60abc, user_id=fe764e185aea4545974fcc05498b6b79, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:17.917350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:17.931621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 895/1796] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:17 2022] PUT /identity/v3/projects/6f18edbbd7d14f6aaebe992d45b60abc/users/fe764e185aea4545974fcc05498b6b79/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:17.943409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9470cd59-068e-4456-a940-7ddd6c158050 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:17.943696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9470cd59-068e-4456-a940-7ddd6c158050 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:17.943972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9470cd59-068e-4456-a940-7ddd6c158050 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:17.973901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9470cd59-068e-4456-a940-7ddd6c158050 None None] MFA Rules not processed for user `fe764e185aea4545974fcc05498b6b79`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:17.975153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9470cd59-068e-4456-a940-7ddd6c158050 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:18.042992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 902/1797] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:17 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:18.060982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:18.063168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:18.125113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32a0c6ca-8a4a-4ffe-8cac-421d15537b65 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:18.128956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd8507a-a786-4ad3-bec1-9968bcc95766 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:18.129558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd8507a-a786-4ad3-bec1-9968bcc95766 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:18.129828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd8507a-a786-4ad3-bec1-9968bcc95766 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:18.130124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd8507a-a786-4ad3-bec1-9968bcc95766 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:18.138789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd8507a-a786-4ad3-bec1-9968bcc95766 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:18.140289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cd8507a-a786-4ad3-bec1-9968bcc95766 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:18.149268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 896/1798] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:18 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:27.668611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9470cd59-068e-4456-a940-7ddd6c158050 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:27.735346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9470cd59-068e-4456-a940-7ddd6c158050 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:27.739997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:27.740628 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:27.740899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:27.741166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] PATH_INFO: `/v3/users/fe764e185aea4545974fcc05498b6b79` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:27.742325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] RBAC: Authorizing `identity:delete_user(user_id=fe764e185aea4545974fcc05498b6b79)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:27.744232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:27.802621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fe764e185aea4545974fcc05498b6b79'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:27.809379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fe764e185aea4545974fcc05498b6b79'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:27.810110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fe764e185aea4545974fcc05498b6b79'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:27.810734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fe764e185aea4545974fcc05498b6b79'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:27.819667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fe764e185aea4545974fcc05498b6b79'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:27.833944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:27.842700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 903/1799] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:27 2022] DELETE /identity/v3/users/fe764e185aea4545974fcc05498b6b79 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:27.844742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:27.845212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:27.845302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:27.865513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:27.866684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:27.927562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 897/1800] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:27.937382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:27.939570 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:28.002959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c607822b-86d7-4780-82a3-a1cada6d6aed admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:28.006698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fba5193-405e-4e2d-9ac7-6af115f34701 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:28.007304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fba5193-405e-4e2d-9ac7-6af115f34701 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.007574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fba5193-405e-4e2d-9ac7-6af115f34701 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.007838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fba5193-405e-4e2d-9ac7-6af115f34701 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.017702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fba5193-405e-4e2d-9ac7-6af115f34701 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:28.019189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fba5193-405e-4e2d-9ac7-6af115f34701 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:28.028849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 904/1801] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:28.398473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:28.400020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:28.415383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83f75eb6-fea7-40b7-93a2-4e4e6853ac4c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:28.433627 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:28.434433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.434744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.435013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] PATH_INFO: `/v3/projects/6f18edbbd7d14f6aaebe992d45b60abc` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.436910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] RBAC: Authorizing `identity:delete_project(project_id=6f18edbbd7d14f6aaebe992d45b60abc)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:28.439252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:28.521657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6f18edbbd7d14f6aaebe992d45b60abc', 'request_id': 'req-2b681575-20e0-48c5-ad8e-b212ff971fae'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:28.538390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6f18edbbd7d14f6aaebe992d45b60abc', 'request_id': 'req-2b681575-20e0-48c5-ad8e-b212ff971fae'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:28.548371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6f18edbbd7d14f6aaebe992d45b60abc', 'request_id': 'req-2b681575-20e0-48c5-ad8e-b212ff971fae'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:28.576134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] The token cache is being invalidate because project 6f18edbbd7d14f6aaebe992d45b60abc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:28.577396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:28.577396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2b681575-20e0-48c5-ad8e-b212ff971fae admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:28.579035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 898/1802] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:28 2022] DELETE /identity/v3/projects/6f18edbbd7d14f6aaebe992d45b60abc => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:28.584902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdec932b-1a45-409d-b10e-08035802cce5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.585188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdec932b-1a45-409d-b10e-08035802cce5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.585459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdec932b-1a45-409d-b10e-08035802cce5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.607121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bdec932b-1a45-409d-b10e-08035802cce5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:28.609366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bdec932b-1a45-409d-b10e-08035802cce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:28.688409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 905/1803] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:28.696482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13395b3f-bad5-4a2b-94de-cf2bb673c2eb None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.696768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13395b3f-bad5-4a2b-94de-cf2bb673c2eb None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.697041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13395b3f-bad5-4a2b-94de-cf2bb673c2eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.717432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-13395b3f-bad5-4a2b-94de-cf2bb673c2eb None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:28.719329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-13395b3f-bad5-4a2b-94de-cf2bb673c2eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:28.728656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 899/1804] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:28.732889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdec932b-1a45-409d-b10e-08035802cce5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:28.743933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdec932b-1a45-409d-b10e-08035802cce5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:28.748754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:28.749429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.749753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.750052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.750706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:28.751007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:28.752628 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:28.766052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 906/1805] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:28.770413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13395b3f-bad5-4a2b-94de-cf2bb673c2eb None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:28.775022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:28.781010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13395b3f-bad5-4a2b-94de-cf2bb673c2eb None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:28.787518 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:28.788207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.788554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.788880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.789737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:28.791869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:28.830087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 900/1806] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:28 2022] POST /identity/v3/projects => generated 373 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:28.831352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:28.842307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-367583b8-bb2a-411b-ba04-05a36779e07c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:28.898576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:28.899186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.899458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.899727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.900356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:28.902311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:28.925864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 907/1807] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:28 2022] POST /identity/v3/users => generated 287 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:28.930406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:28.941291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0f4e6e7-0764-45e5-bc6d-0d77eaff57ae admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:28.945909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:28.946542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.946821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.947091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.947733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:28.949350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:28.959685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 901/1808] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:28.963907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:28.975075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece05d60-c896-45ff-8d0a-645e4a07a2ea admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:28.980483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cc3d31d-7cad-46a3-8f73-7c14b2c233d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:28.981092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3d31d-7cad-46a3-8f73-7c14b2c233d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:28.981358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3d31d-7cad-46a3-8f73-7c14b2c233d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:28.981627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc3d31d-7cad-46a3-8f73-7c14b2c233d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:28.982305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc3d31d-7cad-46a3-8f73-7c14b2c233d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:28.983996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cc3d31d-7cad-46a3-8f73-7c14b2c233d4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:28.994713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 908/1809] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:28.999212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.013717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3d0fde-a56e-4e97-a7d4-ad80ebc55cd5 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:29.019922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:29.020674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.020961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.021202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] PATH_INFO: `/v3/projects/2573e805088142a99e8250d3fc0478db/users/1f06c65d7db443afb919bb6133669355/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.037068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2573e805088142a99e8250d3fc0478db, user_id=1f06c65d7db443afb919bb6133669355, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:29.039905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:29.057383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 902/1810] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:28 2022] PUT /identity/v3/projects/2573e805088142a99e8250d3fc0478db/users/1f06c65d7db443afb919bb6133669355/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:29.068588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e27fca-128a-4ea6-832e-3eec6440b24d None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.068900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e27fca-128a-4ea6-832e-3eec6440b24d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.069195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e27fca-128a-4ea6-832e-3eec6440b24d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.088851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:29.106802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-67e27fca-128a-4ea6-832e-3eec6440b24d None None] MFA Rules not processed for user `1f06c65d7db443afb919bb6133669355`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:29.108068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-67e27fca-128a-4ea6-832e-3eec6440b24d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:29.175364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 909/1811] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:29 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:29.186988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.189036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:29.250731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eb4072-801d-45ac-8d50-2e1077ffd0a9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:29.254893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb5a31e-f7a6-4c0b-8341-2ee784dcd2aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:29.255582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5a31e-f7a6-4c0b-8341-2ee784dcd2aa service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.255975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5a31e-f7a6-4c0b-8341-2ee784dcd2aa service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.256277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb5a31e-f7a6-4c0b-8341-2ee784dcd2aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.267006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb5a31e-f7a6-4c0b-8341-2ee784dcd2aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:29.268463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cb5a31e-f7a6-4c0b-8341-2ee784dcd2aa service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:29.280102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 903/1812] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:29 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:29.315449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e27fca-128a-4ea6-832e-3eec6440b24d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.384377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e27fca-128a-4ea6-832e-3eec6440b24d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:29.388984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:29.389666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.389936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.390230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] PATH_INFO: `/v3/users/1f06c65d7db443afb919bb6133669355` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.391396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1f06c65d7db443afb919bb6133669355)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:29.393305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:29.461557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1f06c65d7db443afb919bb6133669355'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.473278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1f06c65d7db443afb919bb6133669355'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.474050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1f06c65d7db443afb919bb6133669355'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.474665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1f06c65d7db443afb919bb6133669355'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.486713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1f06c65d7db443afb919bb6133669355'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:29.507983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 910/1813] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:29 2022] DELETE /identity/v3/users/1f06c65d7db443afb919bb6133669355 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:29.512734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.513016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.513286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.543344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:29.544923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:29.683322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 904/1814] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:29.801653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:29.803317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:29.838965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:29.963520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5837a75-7a43-4012-b4bb-0f48e07ede57 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:29.975273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-102aa078-b8e3-4312-a5e3-a977ad4397ef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:29.976439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102aa078-b8e3-4312-a5e3-a977ad4397ef service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:29.976903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102aa078-b8e3-4312-a5e3-a977ad4397ef service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:29.977705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-102aa078-b8e3-4312-a5e3-a977ad4397ef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:29.994649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102aa078-b8e3-4312-a5e3-a977ad4397ef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:29.995904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-102aa078-b8e3-4312-a5e3-a977ad4397ef service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:30.011173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 911/1815] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:30.925598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:30.927136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:30.940168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d2c8320-b79c-4f51-8c46-0342e4b71eac None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:30.944894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:30.945650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:30.946149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:30.946663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] PATH_INFO: `/v3/users/948461ec333c4dc49d2fd11f5759ea42` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:30.947642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] RBAC: Authorizing `identity:delete_user(user_id=948461ec333c4dc49d2fd11f5759ea42)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:30.949610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.007228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:INSERT=2,keystone:DELETE=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:31.009931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '948461ec333c4dc49d2fd11f5759ea42'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.019553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '948461ec333c4dc49d2fd11f5759ea42'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.020339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '948461ec333c4dc49d2fd11f5759ea42'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.020960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '948461ec333c4dc49d2fd11f5759ea42'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.033903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '948461ec333c4dc49d2fd11f5759ea42'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.066828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 905/1816] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:30 2022] DELETE /identity/v3/users/948461ec333c4dc49d2fd11f5759ea42 => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:31.071389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.071878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.071965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.092449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:31.093977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.231029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 912/1817] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:31.247595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.251453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.362427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8bb1a22-731f-4ca3-b067-3d9d9b40ffbe admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.373541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-230a6b60-acf5-4949-a995-05fd1d157c56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.374740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a6b60-acf5-4949-a995-05fd1d157c56 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.375022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a6b60-acf5-4949-a995-05fd1d157c56 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.375290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-230a6b60-acf5-4949-a995-05fd1d157c56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.381004 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.383216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.391942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230a6b60-acf5-4949-a995-05fd1d157c56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.397911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2f6ed58-9874-4d46-9e01-494c236e5c8e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.398427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-230a6b60-acf5-4949-a995-05fd1d157c56 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.404619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.405525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.406050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.406659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] PATH_INFO: `/v3/projects/2573e805088142a99e8250d3fc0478db` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.408000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2573e805088142a99e8250d3fc0478db)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.412620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.421235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 906/1818] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:31.548872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2573e805088142a99e8250d3fc0478db', 'request_id': 'req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.561921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2573e805088142a99e8250d3fc0478db', 'request_id': 'req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.575039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2573e805088142a99e8250d3fc0478db', 'request_id': 'req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.594682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] The token cache is being invalidate because project 2573e805088142a99e8250d3fc0478db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:31.594986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.602142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bbf3d63c-9fba-42fd-b69e-d4dff66367e0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:31.610579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 913/1819] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:31 2022] DELETE /identity/v3/projects/2573e805088142a99e8250d3fc0478db => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:31.616118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727fac4-e86b-41dd-8a10-213fcba6de97 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.616507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727fac4-e86b-41dd-8a10-213fcba6de97 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.616785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b727fac4-e86b-41dd-8a10-213fcba6de97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.654670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b727fac4-e86b-41dd-8a10-213fcba6de97 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:31.656447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b727fac4-e86b-41dd-8a10-213fcba6de97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.681634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529caaee-415f-420f-9cfd-6e6eaeead5ad None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.682109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529caaee-415f-420f-9cfd-6e6eaeead5ad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.682385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-529caaee-415f-420f-9cfd-6e6eaeead5ad None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.683749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 914/1820] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:13:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:13:31.708121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d58d77-8b61-4732-8958-5d6d09cbe587 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.708995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d58d77-8b61-4732-8958-5d6d09cbe587 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.709544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d58d77-8b61-4732-8958-5d6d09cbe587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.760048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a2d58d77-8b61-4732-8958-5d6d09cbe587 None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:31.771776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a2d58d77-8b61-4732-8958-5d6d09cbe587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.834061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 907/1821] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:31.839638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94a18b-2d22-4439-9e35-858ec0a55e3c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.840556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94a18b-2d22-4439-9e35-858ec0a55e3c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.841085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae94a18b-2d22-4439-9e35-858ec0a55e3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.857211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:31.891142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ae94a18b-2d22-4439-9e35-858ec0a55e3c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:31.892342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ae94a18b-2d22-4439-9e35-858ec0a55e3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:31.901814 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 908/1822] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 64 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:31.906697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94a18b-2d22-4439-9e35-858ec0a55e3c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.912438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 915/1823] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:31.918233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae94a18b-2d22-4439-9e35-858ec0a55e3c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.927484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.928394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.928755 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.929106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.929859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:31.930289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.932004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:31.946072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 909/1824] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:31.951070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d58d77-8b61-4732-8958-5d6d09cbe587 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:31.967003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d58d77-8b61-4732-8958-5d6d09cbe587 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:31.975202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:31.976093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:31.976395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:31.976833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:31.977574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:31.980051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.021799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 916/1825] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:31 2022] POST /identity/v3/projects => generated 373 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:32.025819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.037925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122c8bc0-63ee-4f2c-8bfa-974f01d7a04d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.116273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=4,keystone:SELECT=95,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:32.132479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.135024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.146365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.147094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.147372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.147645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.148284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.150587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.156855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828cb63-4ad1-415c-a75d-7f306d9756ae admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.185863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.186826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.187106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.187380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] PATH_INFO: `/v3/projects/fdb8dc40956f41938b60febf378f2690` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.189078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=fdb8dc40956f41938b60febf378f2690)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.191041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.217849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 910/1826] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:32 2022] POST /identity/v3/users => generated 287 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:32.218777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.224137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.255663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fdb8dc40956f41938b60febf378f2690', 'request_id': 'req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.264949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fdb8dc40956f41938b60febf378f2690', 'request_id': 'req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.273540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fdb8dc40956f41938b60febf378f2690', 'request_id': 'req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.287075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] The token cache is being invalidate because project fdb8dc40956f41938b60febf378f2690 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:32.287577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.288579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.290258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 917/1827] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:32 2022] DELETE /identity/v3/projects/fdb8dc40956f41938b60febf378f2690 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:32.291486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.293069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.315826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4633591-4941-4ee2-99fe-94928e1723a9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.366484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2803a4f2-2a5a-4956-ad2d-3c40c0ed25fc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.371172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.371777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.372049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.372315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.372951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.375073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.381756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.382376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.382685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.382952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.384851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.385244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 918/1828] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:32 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:32.386202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.389626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.396780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.398399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.401989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af5b577c-a4b2-40fb-828d-08ab9c7dfc09 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.406785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.407386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.407657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.407924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] PATH_INFO: `/v3/users/990eae56faa54e7a9fcdcff8f4906747` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.409012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 911/1829] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:32.409117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] RBAC: Authorizing `identity:delete_user(user_id=990eae56faa54e7a9fcdcff8f4906747)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.409797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.410955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.420190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d214cda8-d0b8-4214-b65e-0a70a9bbe90a service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.424799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.425396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.425669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.425932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.426656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.428234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.440142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 912/1830] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:32 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:32.444579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.448245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:INSERT=2,keystone:DELETE=11 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:32.458084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5dfb53-5ce8-4b8f-8441-de667b779d19 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.463610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84297f7c-8a35-4add-b81b-057e4bdb33dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.464408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84297f7c-8a35-4add-b81b-057e4bdb33dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.464685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84297f7c-8a35-4add-b81b-057e4bdb33dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.464959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84297f7c-8a35-4add-b81b-057e4bdb33dd admin admin] PATH_INFO: `/v3/projects/10095a8153824264ae08945d9f273d16/users/ae138369efde4dc798360a54b95dd076/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.472440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '990eae56faa54e7a9fcdcff8f4906747'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.479371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '990eae56faa54e7a9fcdcff8f4906747'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.480019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '990eae56faa54e7a9fcdcff8f4906747'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.480565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84297f7c-8a35-4add-b81b-057e4bdb33dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=10095a8153824264ae08945d9f273d16, user_id=ae138369efde4dc798360a54b95dd076, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.480699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '990eae56faa54e7a9fcdcff8f4906747'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.483366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84297f7c-8a35-4add-b81b-057e4bdb33dd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.489400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '990eae56faa54e7a9fcdcff8f4906747'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:32.497986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 913/1831] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:32 2022] PUT /identity/v3/projects/10095a8153824264ae08945d9f273d16/users/ae138369efde4dc798360a54b95dd076/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:32.507288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 919/1832] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:32 2022] DELETE /identity/v3/users/990eae56faa54e7a9fcdcff8f4906747 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:32.508693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.509043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.509314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.545342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] MFA Rules not processed for user `ae138369efde4dc798360a54b95dd076`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:32.546804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.560994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:32.645520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 914/1833] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:32 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:32.657244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.659117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.774369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b124bcd-2870-4571-badf-cab90fb37c6b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.785118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb395d7a-3302-4420-9e42-b8e05853be9e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.785753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb395d7a-3302-4420-9e42-b8e05853be9e service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.786042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb395d7a-3302-4420-9e42-b8e05853be9e service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.786313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb395d7a-3302-4420-9e42-b8e05853be9e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.806324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb395d7a-3302-4420-9e42-b8e05853be9e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.806324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb395d7a-3302-4420-9e42-b8e05853be9e service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:32.818789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 920/1834] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:32 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:32.871588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:32.873056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:32.947253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e79e21d-aaee-4be0-9c33-50ae0fe23261 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:32.951987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:32.952828 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:32.953147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:32.953394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] PATH_INFO: `/v3/projects/e65c06949e1c4d02ada8f42dc10c2ed3` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:32.954587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e65c06949e1c4d02ada8f42dc10c2ed3)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:32.956491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.004903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e65c06949e1c4d02ada8f42dc10c2ed3', 'request_id': 'req-0b99dde0-b0d3-4330-a972-898048d90482'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.013897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e65c06949e1c4d02ada8f42dc10c2ed3', 'request_id': 'req-0b99dde0-b0d3-4330-a972-898048d90482'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.020966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e65c06949e1c4d02ada8f42dc10c2ed3', 'request_id': 'req-0b99dde0-b0d3-4330-a972-898048d90482'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.031430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] The token cache is being invalidate because project e65c06949e1c4d02ada8f42dc10c2ed3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:33.031719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.032372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0b99dde0-b0d3-4330-a972-898048d90482 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:33.034069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 915/1835] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:32 2022] DELETE /identity/v3/projects/e65c06949e1c4d02ada8f42dc10c2ed3 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:33.038985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0baa2c-b157-4107-9600-fbbd1e8f2d4e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.039341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0baa2c-b157-4107-9600-fbbd1e8f2d4e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.039712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a0baa2c-b157-4107-9600-fbbd1e8f2d4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.060678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8a0baa2c-b157-4107-9600-fbbd1e8f2d4e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:33.061980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8a0baa2c-b157-4107-9600-fbbd1e8f2d4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.130693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 921/1836] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:33.139034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dacea-5d59-4c10-bd50-3353761527a0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.139554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dacea-5d59-4c10-bd50-3353761527a0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.140043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dacea-5d59-4c10-bd50-3353761527a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.166495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-406dacea-5d59-4c10-bd50-3353761527a0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:33.167754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-406dacea-5d59-4c10-bd50-3353761527a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.176707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 916/1837] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:33.180860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0baa2c-b157-4107-9600-fbbd1e8f2d4e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.194316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a0baa2c-b157-4107-9600-fbbd1e8f2d4e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.200671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.201564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.201863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.202186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.203250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:33.203557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.205483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.231082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 922/1838] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:33.235603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406dacea-5d59-4c10-bd50-3353761527a0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.249383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406dacea-5d59-4c10-bd50-3353761527a0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.262854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.263488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.263768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.264388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.265001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.267368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.318100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 917/1839] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/projects => generated 387 bytes in 83 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:33.323550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.333440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd373d87-ea0e-4370-add0-0f56dbd5fbf1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.387555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=6,keystone:SELECT=93,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:33.417952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.418638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.418919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.419186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.419820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.421959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.446775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 923/1840] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/users => generated 294 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:33.448237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.458793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7e8282-1ea2-4a5a-9409-d650a9864e19 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.463298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.463900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.464175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.464441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.465082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.467057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.477765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 918/1841] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:33.481837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.492503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5254825-8e16-4ef9-8fb5-d3bda7581ff2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.498426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0958db-bb43-4d05-8bf3-e51eb6d5f30e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.498919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0958db-bb43-4d05-8bf3-e51eb6d5f30e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.499212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0958db-bb43-4d05-8bf3-e51eb6d5f30e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.499494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0958db-bb43-4d05-8bf3-e51eb6d5f30e admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.500138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0958db-bb43-4d05-8bf3-e51eb6d5f30e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.501731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0958db-bb43-4d05-8bf3-e51eb6d5f30e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.512302 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 924/1842] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:33.516609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.522128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:33.528662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70af2a9-1b62-4392-b271-309e1cdbc4b0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.534647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.535514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.535846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.536333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] PATH_INFO: `/v3/projects/cc3f2e34f9ea415d8c541c686ab2f6f0/users/ddc87bd5d7014a1e9390c1d834be660d/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.551323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc3f2e34f9ea415d8c541c686ab2f6f0, user_id=ddc87bd5d7014a1e9390c1d834be660d, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.553987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.570106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 919/1843] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:33 2022] PUT /identity/v3/projects/cc3f2e34f9ea415d8c541c686ab2f6f0/users/ddc87bd5d7014a1e9390c1d834be660d/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:33.582920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e48026b-c3b5-4aa1-bc2b-cabf2a056dd8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.583305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e48026b-c3b5-4aa1-bc2b-cabf2a056dd8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.583719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e48026b-c3b5-4aa1-bc2b-cabf2a056dd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.616338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7e48026b-c3b5-4aa1-bc2b-cabf2a056dd8 None None] MFA Rules not processed for user `ddc87bd5d7014a1e9390c1d834be660d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:33.617533 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7e48026b-c3b5-4aa1-bc2b-cabf2a056dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:33.683218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 925/1844] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:33.687981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.754321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90746dc7-d504-4015-90c3-0f43168be19d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.763007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.763840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.764239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.764529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.765342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.767412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.801003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 920/1845] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/projects => generated 387 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:33.804932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e48026b-c3b5-4aa1-bc2b-cabf2a056dd8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.816726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e48026b-c3b5-4aa1-bc2b-cabf2a056dd8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.888998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.889622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.890108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.890931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.891624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.893795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.918779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 926/1846] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:33 2022] POST /identity/v3/users => generated 300 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:33.923553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.952621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e20ec8-ad27-4d7a-9755-e1772e2bccf6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:33.958898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:33.959733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:33.960056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:33.960326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:33.961095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:33.964132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:33.978204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 921/1847] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:33 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:33.982521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:33.994540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d665a0ff-f6bc-4545-8817-f92f2748e548 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.009197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.009197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.009197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.009197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] PATH_INFO: `/v3/projects/2145fa65312343d59be004b4d7771276/users/71947fa8a0a14df8a007f3d9b1e1bef3/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.026876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2145fa65312343d59be004b4d7771276, user_id=71947fa8a0a14df8a007f3d9b1e1bef3, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.029490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.044151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 927/1848] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:33 2022] PUT /identity/v3/projects/2145fa65312343d59be004b4d7771276/users/71947fa8a0a14df8a007f3d9b1e1bef3/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:34.048969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.059006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9199dac5-8bc2-4b40-b1a1-b3d6f437fb78 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.114899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107801e1-2b7e-4cef-96ec-28c60dcf7ca9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.115504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107801e1-2b7e-4cef-96ec-28c60dcf7ca9 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.115774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107801e1-2b7e-4cef-96ec-28c60dcf7ca9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.116055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107801e1-2b7e-4cef-96ec-28c60dcf7ca9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.116699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107801e1-2b7e-4cef-96ec-28c60dcf7ca9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.118370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-107801e1-2b7e-4cef-96ec-28c60dcf7ca9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.128258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 922/1849] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:34 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:34.132528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.143099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cf4476-d72d-4095-8599-ee61424e25e4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.147813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.148462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.148745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.149011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] PATH_INFO: `/v3/projects/2145fa65312343d59be004b4d7771276/users/71947fa8a0a14df8a007f3d9b1e1bef3/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.154343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2145fa65312343d59be004b4d7771276, user_id=71947fa8a0a14df8a007f3d9b1e1bef3, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.157041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.159359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:34.169526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 928/1850] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:34 2022] PUT /identity/v3/projects/2145fa65312343d59be004b4d7771276/users/71947fa8a0a14df8a007f3d9b1e1bef3/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:34.180514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c5e175-94e4-4ff8-8699-2ea6b99842a6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.180796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c5e175-94e4-4ff8-8699-2ea6b99842a6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.181070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c5e175-94e4-4ff8-8699-2ea6b99842a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.209508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a4c5e175-94e4-4ff8-8699-2ea6b99842a6 None None] MFA Rules not processed for user `71947fa8a0a14df8a007f3d9b1e1bef3`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:34.210769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a4c5e175-94e4-4ff8-8699-2ea6b99842a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:34.251549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:34.287321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 923/1851] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:34.299156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.301019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:34.360361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d27d1151-28e8-4030-a4bc-94e1a4f74de2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.364147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63003385-b6cf-4346-b272-ae223f2307e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.364733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63003385-b6cf-4346-b272-ae223f2307e9 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.364983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63003385-b6cf-4346-b272-ae223f2307e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.365242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63003385-b6cf-4346-b272-ae223f2307e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.374273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63003385-b6cf-4346-b272-ae223f2307e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.375699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63003385-b6cf-4346-b272-ae223f2307e9 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.385323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 929/1852] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:34 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:34.457611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c5e175-94e4-4ff8-8699-2ea6b99842a6 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.541112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c5e175-94e4-4ff8-8699-2ea6b99842a6 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.546056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.546652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.547418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.547418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] PATH_INFO: `/v3/users/ddc87bd5d7014a1e9390c1d834be660d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.548340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddc87bd5d7014a1e9390c1d834be660d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.550305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.610405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddc87bd5d7014a1e9390c1d834be660d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.617580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddc87bd5d7014a1e9390c1d834be660d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.618294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddc87bd5d7014a1e9390c1d834be660d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.618910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddc87bd5d7014a1e9390c1d834be660d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.627760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddc87bd5d7014a1e9390c1d834be660d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:34.646829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 924/1853] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:34 2022] DELETE /identity/v3/users/ddc87bd5d7014a1e9390c1d834be660d => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:34.651243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.651529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.651926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.672116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:34.673345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:34.733218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 930/1854] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:34.743058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:34.744491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:34.807288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-151e03e7-7e22-445f-86e0-fa5c6b2c9d7d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:34.811364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:34.812020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:34.812300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:34.812563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:34.819582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:34.822120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:34.824030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:34.834461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 925/1855] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:35.152556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:35.154997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:35.170680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89d55149-3ee9-44b6-bd89-605a85f48221 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:35.178109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:35.178927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:35.179040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:35.179337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] PATH_INFO: `/v3/projects/cc3f2e34f9ea415d8c541c686ab2f6f0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:35.180527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc3f2e34f9ea415d8c541c686ab2f6f0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:35.182593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:35.235928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc3f2e34f9ea415d8c541c686ab2f6f0', 'request_id': 'req-292ea598-cdcf-4549-9f5c-6546138befee'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.244907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc3f2e34f9ea415d8c541c686ab2f6f0', 'request_id': 'req-292ea598-cdcf-4549-9f5c-6546138befee'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.251615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc3f2e34f9ea415d8c541c686ab2f6f0', 'request_id': 'req-292ea598-cdcf-4549-9f5c-6546138befee'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.270213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] The token cache is being invalidate because project cc3f2e34f9ea415d8c541c686ab2f6f0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:35.270503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.271164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.272683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 931/1856] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:35 2022] DELETE /identity/v3/projects/cc3f2e34f9ea415d8c541c686ab2f6f0 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:35.276652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:35.286447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7d9f15-886c-4477-b33d-2536e4eb21b1 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:35.341480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:35.342438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:35.342726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:35.343008 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] PATH_INFO: `/v3/users/71947fa8a0a14df8a007f3d9b1e1bef3` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:35.344275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=71947fa8a0a14df8a007f3d9b1e1bef3)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:35.346273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:35.414475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '71947fa8a0a14df8a007f3d9b1e1bef3'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.422240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '71947fa8a0a14df8a007f3d9b1e1bef3'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.422935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '71947fa8a0a14df8a007f3d9b1e1bef3'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.423577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '71947fa8a0a14df8a007f3d9b1e1bef3'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.433825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-74b5c3bc-83ca-4855-9911-1d3b3eef62c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '71947fa8a0a14df8a007f3d9b1e1bef3'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:35.452168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 926/1857] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:35 2022] DELETE /identity/v3/users/71947fa8a0a14df8a007f3d9b1e1bef3 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:35.841577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:35.844394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:35.857985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ea598-cdcf-4549-9f5c-6546138befee admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:35.881777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:35.941509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:35.942199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:35.942478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:35.942750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] PATH_INFO: `/v3/projects/2145fa65312343d59be004b4d7771276` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:35.943907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=2145fa65312343d59be004b4d7771276)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:35.945821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:35.995445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2145fa65312343d59be004b4d7771276', 'request_id': 'req-5d072a63-7f2b-43c7-a100-53c854e741eb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:36.005493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2145fa65312343d59be004b4d7771276', 'request_id': 'req-5d072a63-7f2b-43c7-a100-53c854e741eb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:36.012634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2145fa65312343d59be004b4d7771276', 'request_id': 'req-5d072a63-7f2b-43c7-a100-53c854e741eb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:36.023359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] The token cache is being invalidate because project 2145fa65312343d59be004b4d7771276 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:36.023594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:36.024268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5d072a63-7f2b-43c7-a100-53c854e741eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:36.025747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 932/1858] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:35 2022] DELETE /identity/v3/projects/2145fa65312343d59be004b4d7771276 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:36.031065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55c512a-4fc7-48ab-9665-19361f28f971 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.031260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55c512a-4fc7-48ab-9665-19361f28f971 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.031532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55c512a-4fc7-48ab-9665-19361f28f971 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.052596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f55c512a-4fc7-48ab-9665-19361f28f971 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:36.053801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f55c512a-4fc7-48ab-9665-19361f28f971 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:36.112263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 927/1859] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:36.120258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6034ec1-2571-45d3-a606-cc222b04058b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.120441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6034ec1-2571-45d3-a606-cc222b04058b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.120768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6034ec1-2571-45d3-a606-cc222b04058b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.141657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d6034ec1-2571-45d3-a606-cc222b04058b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:36.142906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d6034ec1-2571-45d3-a606-cc222b04058b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:36.151456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 933/1860] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:36.156061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55c512a-4fc7-48ab-9665-19361f28f971 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.165857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55c512a-4fc7-48ab-9665-19361f28f971 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.170694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.171344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.171622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.171891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.172538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:36.172839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.174488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.186565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 928/1861] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:36.191215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6034ec1-2571-45d3-a606-cc222b04058b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.200796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6034ec1-2571-45d3-a606-cc222b04058b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.205830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.206466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.206749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.207018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.207668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.209593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.239625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 934/1862] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:36 2022] POST /identity/v3/projects => generated 409 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:36.243823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.249213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:36.255137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51a4a097-5a6e-4616-aaf6-064f1d6c9f7d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.320799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.321452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.321657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.322148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.322585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.324509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.353459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 929/1863] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:36 2022] POST /identity/v3/users => generated 305 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:36.355314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.365400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ebc0a2-1111-40bd-b5fe-300f1701bdc3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.370105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.370767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.370981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.371245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.371884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.373487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.385617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 935/1864] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:36.390035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.403338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb85dd63-8649-4940-9380-c9bdfc2247e1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.408341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c35f005e-2532-4043-8b38-5a49d67fb31d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.409030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f005e-2532-4043-8b38-5a49d67fb31d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.409315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f005e-2532-4043-8b38-5a49d67fb31d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.409587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c35f005e-2532-4043-8b38-5a49d67fb31d admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.410445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f005e-2532-4043-8b38-5a49d67fb31d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.412073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c35f005e-2532-4043-8b38-5a49d67fb31d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.424517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 930/1865] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:36.428656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.439639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456e9a3c-0c2c-425a-99b1-4543754de4fe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.444405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.444953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.445231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.445507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] PATH_INFO: `/v3/projects/24f36f040e3e4cd6bbc0cf0703a4cba3/users/52230ba322754f16b37c8fdc1b6ac119/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.472699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=24f36f040e3e4cd6bbc0cf0703a4cba3, user_id=52230ba322754f16b37c8fdc1b6ac119, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.475426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.490105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 936/1866] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:36 2022] PUT /identity/v3/projects/24f36f040e3e4cd6bbc0cf0703a4cba3/users/52230ba322754f16b37c8fdc1b6ac119/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:36.501871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b283fb12-a4f7-4fea-bbb2-a2d042bc65dc None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.502189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b283fb12-a4f7-4fea-bbb2-a2d042bc65dc None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.502469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b283fb12-a4f7-4fea-bbb2-a2d042bc65dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.530423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b283fb12-a4f7-4fea-bbb2-a2d042bc65dc None None] MFA Rules not processed for user `52230ba322754f16b37c8fdc1b6ac119`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:36.532271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b283fb12-a4f7-4fea-bbb2-a2d042bc65dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:36.603989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 931/1867] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:36 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:36.608664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.660320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:36.676075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b1dabe-b43d-4ccf-9785-630950fe4e4f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.681264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.681869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.682184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.682456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.683081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.685001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.714809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 937/1868] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:36 2022] POST /identity/v3/projects => generated 409 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:36.719596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b283fb12-a4f7-4fea-bbb2-a2d042bc65dc None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.731445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b283fb12-a4f7-4fea-bbb2-a2d042bc65dc None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.792233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.792883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.793161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.793439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.794103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.796153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.815888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:36.822185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 932/1869] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:36 2022] POST /identity/v3/users => generated 311 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:36.826046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.837849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fca9cf7c-5ae6-47d9-9133-651dbb7e6772 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.844378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.845052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.845266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.847097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.847603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.849385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.865579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 938/1870] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:36.866712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.878862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1fe619a-7451-4824-898a-5806b61fa4c6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:36.885095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:36.885899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:36.886236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:36.886508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] PATH_INFO: `/v3/projects/8fe4fadd97494b8eb82e1353fec6dcd9/users/84c7a324ef354f2b86d058dce9e5cf01/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:36.907628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fe4fadd97494b8eb82e1353fec6dcd9, user_id=84c7a324ef354f2b86d058dce9e5cf01, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:36.910701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:36.934061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 933/1871] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:36 2022] PUT /identity/v3/projects/8fe4fadd97494b8eb82e1353fec6dcd9/users/84c7a324ef354f2b86d058dce9e5cf01/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:36.935398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:36.946685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaeb1ebe-cab6-4ef5-96a8-6967fcea2095 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:37.069112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fa22a8-6100-4ceb-a950-02cf31b285aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:37.069731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fa22a8-6100-4ceb-a950-02cf31b285aa admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:37.070028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fa22a8-6100-4ceb-a950-02cf31b285aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:37.070422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fa22a8-6100-4ceb-a950-02cf31b285aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:37.070945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fa22a8-6100-4ceb-a950-02cf31b285aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:37.072572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fa22a8-6100-4ceb-a950-02cf31b285aa admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:37.086049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 939/1872] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:36 2022] GET /identity/v3/roles => generated 1420 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:37.088328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:37.127187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71b7a3f-a7f5-48e1-9ca4-0e272c557610 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:37.146775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:37.148156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:37.148647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:37.149123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] PATH_INFO: `/v3/projects/8fe4fadd97494b8eb82e1353fec6dcd9/users/84c7a324ef354f2b86d058dce9e5cf01/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:37.151304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fe4fadd97494b8eb82e1353fec6dcd9, user_id=84c7a324ef354f2b86d058dce9e5cf01, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:37.154499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:37.169711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 934/1873] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:37 2022] PUT /identity/v3/projects/8fe4fadd97494b8eb82e1353fec6dcd9/users/84c7a324ef354f2b86d058dce9e5cf01/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:37.180189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff89666-6bb9-4873-aba7-78c850b06099 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:37.180475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff89666-6bb9-4873-aba7-78c850b06099 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:37.180747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff89666-6bb9-4873-aba7-78c850b06099 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:37.220045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0ff89666-6bb9-4873-aba7-78c850b06099 None None] MFA Rules not processed for user `84c7a324ef354f2b86d058dce9e5cf01`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:37.222118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0ff89666-6bb9-4873-aba7-78c850b06099 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:37.323817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 940/1874] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:37 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 145 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:37.339313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:37.341173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:37.406039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20dbfae-f3e6-4596-b047-ac04fe171618 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:37.410279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:37.410965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:37.411368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:37.411622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:37.426335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:37.427784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:37.437947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 935/1875] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:37 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:37.665242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff89666-6bb9-4873-aba7-78c850b06099 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:37.688725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff89666-6bb9-4873-aba7-78c850b06099 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:37.695413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd59fdf3-791f-4146-b6d9-bc3c2d36e198 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:37.696867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd59fdf3-791f-4146-b6d9-bc3c2d36e198 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:37.697310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd59fdf3-791f-4146-b6d9-bc3c2d36e198 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:37.697766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd59fdf3-791f-4146-b6d9-bc3c2d36e198 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:37.743123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:37.765483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd59fdf3-791f-4146-b6d9-bc3c2d36e198 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:37.766988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd59fdf3-791f-4146-b6d9-bc3c2d36e198 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:37.776876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 941/1876] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:37 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:41.293790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:41.300070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:41.370184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06d2b18b-7450-4b74-a2bb-8fb78be891ae service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:41.374575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:41.375257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:41.375528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:41.375794 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] PATH_INFO: `/v3/users/1202a5b40a1d4c52bf5a8fe965f5f35e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:41.376942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1202a5b40a1d4c52bf5a8fe965f5f35e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:41.379048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:41.429391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1202a5b40a1d4c52bf5a8fe965f5f35e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:41.436748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1202a5b40a1d4c52bf5a8fe965f5f35e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:41.437766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1202a5b40a1d4c52bf5a8fe965f5f35e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:41.438598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1202a5b40a1d4c52bf5a8fe965f5f35e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:41.447408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1202a5b40a1d4c52bf5a8fe965f5f35e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:41.452007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:41.472178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 936/1877] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:41 2022] DELETE /identity/v3/users/1202a5b40a1d4c52bf5a8fe965f5f35e => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:41.476920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:41.477464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:41.477553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:41.498063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:41.499463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:41.576578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 942/1878] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:41.584659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:41.586200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:41.658241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e8fc28-5773-4322-bcb1-addf64844650 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:41.661320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:41.661935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:41.662258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:41.662527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:41.672522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:41.673964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:41.683515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 937/1879] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:42.075246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:42.076650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:42.089359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e9396b-3397-47c7-92cb-b9457ecc7405 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:42.095003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:42.095650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:42.096094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:42.096369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] PATH_INFO: `/v3/projects/3aa3d6d0a4b14baab0999c7c69f44ea4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:42.098153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] RBAC: Authorizing `identity:delete_project(project_id=3aa3d6d0a4b14baab0999c7c69f44ea4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:42.100167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:42.170265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3aa3d6d0a4b14baab0999c7c69f44ea4', 'request_id': 'req-5b77c302-3acc-496f-9b37-dacb42d7154e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.180822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3aa3d6d0a4b14baab0999c7c69f44ea4', 'request_id': 'req-5b77c302-3acc-496f-9b37-dacb42d7154e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.188193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3aa3d6d0a4b14baab0999c7c69f44ea4', 'request_id': 'req-5b77c302-3acc-496f-9b37-dacb42d7154e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.198707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] The token cache is being invalidate because project 3aa3d6d0a4b14baab0999c7c69f44ea4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:42.199060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.199765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.201330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 943/1880] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:42 2022] DELETE /identity/v3/projects/3aa3d6d0a4b14baab0999c7c69f44ea4 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:42.205472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:42.215074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f209e24-ef69-426f-9ab3-61017b571d41 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:42.276731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:42.277379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:42.277649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:42.277917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] PATH_INFO: `/v3/users/1610a4fe00884d379f7a2f9111d8977d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:42.279209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=1610a4fe00884d379f7a2f9111d8977d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:42.281560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:42.333456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1610a4fe00884d379f7a2f9111d8977d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.346896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1610a4fe00884d379f7a2f9111d8977d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.347606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1610a4fe00884d379f7a2f9111d8977d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.348200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1610a4fe00884d379f7a2f9111d8977d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.359410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-48935610-0ce8-4037-ab43-189c25d1aa1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1610a4fe00884d379f7a2f9111d8977d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.378632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 938/1881] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:42 2022] DELETE /identity/v3/users/1610a4fe00884d379f7a2f9111d8977d => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:42.786868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:42.788311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:42.807850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b77c302-3acc-496f-9b37-dacb42d7154e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:42.867377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:42.868013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:42.868285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:42.868552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] PATH_INFO: `/v3/projects/64c70e18c9a1496cbb96e0087254c5db` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:42.869652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=64c70e18c9a1496cbb96e0087254c5db)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:42.871572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:42.878201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:42.923821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '64c70e18c9a1496cbb96e0087254c5db', 'request_id': 'req-b707ab4c-a28c-4247-aad3-9072b41f91a1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.932182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '64c70e18c9a1496cbb96e0087254c5db', 'request_id': 'req-b707ab4c-a28c-4247-aad3-9072b41f91a1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.938824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '64c70e18c9a1496cbb96e0087254c5db', 'request_id': 'req-b707ab4c-a28c-4247-aad3-9072b41f91a1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.949192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] The token cache is being invalidate because project 64c70e18c9a1496cbb96e0087254c5db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:42.949475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.950128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b707ab4c-a28c-4247-aad3-9072b41f91a1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:42.953885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 944/1882] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:42 2022] DELETE /identity/v3/projects/64c70e18c9a1496cbb96e0087254c5db => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:42.958869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efa261d-5128-40ba-a987-f9abf533393c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:42.959606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efa261d-5128-40ba-a987-f9abf533393c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:42.959606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7efa261d-5128-40ba-a987-f9abf533393c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:42.973668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=12,keystone:SELECT=87,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:42.983185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7efa261d-5128-40ba-a987-f9abf533393c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:42.985855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7efa261d-5128-40ba-a987-f9abf533393c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:43.057153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 939/1883] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:43.065564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3a5fa-c2fa-4599-8ff7-1e8771cc5d02 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.067220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3a5fa-c2fa-4599-8ff7-1e8771cc5d02 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.067643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a3a5fa-c2fa-4599-8ff7-1e8771cc5d02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.097788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c5a3a5fa-c2fa-4599-8ff7-1e8771cc5d02 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:43.099388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c5a3a5fa-c2fa-4599-8ff7-1e8771cc5d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:43.111448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 945/1884] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:43.116065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efa261d-5128-40ba-a987-f9abf533393c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.127469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7efa261d-5128-40ba-a987-f9abf533393c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.132604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.133328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.133691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.134160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.134859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:43.135249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.136944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.152661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 940/1885] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:43.157326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3a5fa-c2fa-4599-8ff7-1e8771cc5d02 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.168347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a3a5fa-c2fa-4599-8ff7-1e8771cc5d02 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.173431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.174268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.174638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.175001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.175740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.178254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.236620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.252644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 946/1886] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:43 2022] POST /identity/v3/projects => generated 379 bytes in 79 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:43.252994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c1bb9f-5925-4cdf-8aba-974716916be6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.359319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.360901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.361512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.361973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.362966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.365111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.398904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 941/1887] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:43 2022] POST /identity/v3/users => generated 290 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:43.403317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.414812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-554a2410-0f06-4f10-a688-a970ef93bc36 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.420562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.421326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.421440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.421780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.422943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.424344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.454228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 947/1888] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:43 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:43.455980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.473693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44619f42-e49c-4ce8-9710-2c22a656962b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.479965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0fcec3c-1546-4b6f-9590-da5106673e9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.480830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fcec3c-1546-4b6f-9590-da5106673e9d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.481317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fcec3c-1546-4b6f-9590-da5106673e9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.481587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0fcec3c-1546-4b6f-9590-da5106673e9d admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.482272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fcec3c-1546-4b6f-9590-da5106673e9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.483861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0fcec3c-1546-4b6f-9590-da5106673e9d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.498590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 942/1889] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:43 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:43.503105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.513472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8f910a8-23e8-4d00-97f7-1b121464dce3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.518671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.519326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.519601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.519870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] PATH_INFO: `/v3/projects/65dafa5bd1fa4ada875c3102789d6b30/users/ac7c610a831a41e59614a9aa2b703b89/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.534394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] RBAC: Authorizing `identity:create_grant(project_id=65dafa5bd1fa4ada875c3102789d6b30, user_id=ac7c610a831a41e59614a9aa2b703b89, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.537040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.551516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 948/1890] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:43 2022] PUT /identity/v3/projects/65dafa5bd1fa4ada875c3102789d6b30/users/ac7c610a831a41e59614a9aa2b703b89/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:43.563070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db56fa-d276-496d-a208-00fde522e33c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.563352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db56fa-d276-496d-a208-00fde522e33c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.563607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31db56fa-d276-496d-a208-00fde522e33c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.594235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-31db56fa-d276-496d-a208-00fde522e33c None None] MFA Rules not processed for user `ac7c610a831a41e59614a9aa2b703b89`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:43.595499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-31db56fa-d276-496d-a208-00fde522e33c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:43.674884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 943/1891] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:43 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:43.679527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.768771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c89d72-51cd-4b18-8abc-67e73d03d786 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.773919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.774688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.774959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.775226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.775894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.777824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.812268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 949/1892] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:43 2022] POST /identity/v3/projects => generated 379 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:43.814261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db56fa-d276-496d-a208-00fde522e33c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.820037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:43.826908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31db56fa-d276-496d-a208-00fde522e33c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.892429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.893054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.893327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.893594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.894271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.896210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.920836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 944/1893] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:43 2022] POST /identity/v3/users => generated 296 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:43.925358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.935941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb018c7f-2442-4122-b119-ac97e81b19a4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.941615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.942289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.942705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.942981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.943619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.945238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:43.960222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 950/1894] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:43 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:43.961872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:43.973003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20049d9-deb6-488a-880a-ac9bba75311e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:43.977899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:43.978598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:43.978875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:43.979147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] PATH_INFO: `/v3/projects/0a2b3767ba564071bef109e40a567777/users/a195a80db0a94235bc34ec1bc06f0d8a/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:43.994582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a2b3767ba564071bef109e40a567777, user_id=a195a80db0a94235bc34ec1bc06f0d8a, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:43.997222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:44.011939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 945/1895] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:43 2022] PUT /identity/v3/projects/0a2b3767ba564071bef109e40a567777/users/a195a80db0a94235bc34ec1bc06f0d8a/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:44.016303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:44.023535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:44.027271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04ede677-485c-42f8-82a5-c759255b895b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:44.087759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4d77139-c64f-4398-8922-04f00cc743e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:44.088364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d77139-c64f-4398-8922-04f00cc743e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:44.088635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d77139-c64f-4398-8922-04f00cc743e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:44.088907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4d77139-c64f-4398-8922-04f00cc743e1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:44.089540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d77139-c64f-4398-8922-04f00cc743e1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:44.091168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4d77139-c64f-4398-8922-04f00cc743e1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:44.101206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 951/1896] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:44 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:44.107273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:44.118285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edbc0f6-93ae-4a12-9e2a-570c590e5cbe admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:44.123911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:44.124605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:44.125104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:44.125379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] PATH_INFO: `/v3/projects/0a2b3767ba564071bef109e40a567777/users/a195a80db0a94235bc34ec1bc06f0d8a/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:44.127679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a2b3767ba564071bef109e40a567777, user_id=a195a80db0a94235bc34ec1bc06f0d8a, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:44.130372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:44.142186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 946/1897] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:44 2022] PUT /identity/v3/projects/0a2b3767ba564071bef109e40a567777/users/a195a80db0a94235bc34ec1bc06f0d8a/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:44.153142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb503b3-fff1-47f8-bcc6-f17b61cb5546 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:44.153425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb503b3-fff1-47f8-bcc6-f17b61cb5546 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:44.153718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bb503b3-fff1-47f8-bcc6-f17b61cb5546 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:44.184141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4bb503b3-fff1-47f8-bcc6-f17b61cb5546 None None] MFA Rules not processed for user `a195a80db0a94235bc34ec1bc06f0d8a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:44.185290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4bb503b3-fff1-47f8-bcc6-f17b61cb5546 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:44.260858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 952/1898] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:44 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:44.273698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:44.275637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:44.337137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-526d99ef-85eb-4438-9490-115b845fdbf6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:44.340925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:44.341658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:44.341848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:44.342114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:44.355426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:44.357923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:44.369700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 947/1899] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:44 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:44.456791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb503b3-fff1-47f8-bcc6-f17b61cb5546 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:44.466701 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bb503b3-fff1-47f8-bcc6-f17b61cb5546 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:44.470719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d3a07f6-3ef8-4756-97c6-0a42a48f8c74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:44.471385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3a07f6-3ef8-4756-97c6-0a42a48f8c74 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:44.471720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3a07f6-3ef8-4756-97c6-0a42a48f8c74 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:44.472051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d3a07f6-3ef8-4756-97c6-0a42a48f8c74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:44.538126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3a07f6-3ef8-4756-97c6-0a42a48f8c74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:44.539588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d3a07f6-3ef8-4756-97c6-0a42a48f8c74 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:44.556435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 953/1900] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:44 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:48.035382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:48.036959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:48.099936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:48.104620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81fdb1b8-56f0-4b8e-a131-d810cd090ca6 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:48.110564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:48.111200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:48.111475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:48.111747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] PATH_INFO: `/v3/users/ae138369efde4dc798360a54b95dd076` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:48.112865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ae138369efde4dc798360a54b95dd076)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:48.114826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:48.166177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ae138369efde4dc798360a54b95dd076'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.173697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ae138369efde4dc798360a54b95dd076'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.174411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ae138369efde4dc798360a54b95dd076'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.175057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ae138369efde4dc798360a54b95dd076'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.185190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ae138369efde4dc798360a54b95dd076'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.204453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 948/1901] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:48 2022] DELETE /identity/v3/users/ae138369efde4dc798360a54b95dd076 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:48.208956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:48.209429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:48.209563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:48.230318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:48.231545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:48.251281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:48.296154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 954/1902] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:48.306905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:48.308408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:48.380147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed78f68b-2175-4ada-829f-ea05c57f19d8 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:48.383785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:48.384485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:48.384817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:48.385148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:48.394468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:48.395909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:48.405914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 949/1903] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:48.721566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:48.723088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:48.736884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2798b8b9-7ac7-4d71-ada0-2c1cd1c15a5a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:48.742318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:48.743067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:48.743465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:48.744890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] PATH_INFO: `/v3/projects/10095a8153824264ae08945d9f273d16` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:48.745363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=10095a8153824264ae08945d9f273d16)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:48.747624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:48.836607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '10095a8153824264ae08945d9f273d16', 'request_id': 'req-29c39ad8-e040-4d83-985c-9a7e555aae8b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.857219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '10095a8153824264ae08945d9f273d16', 'request_id': 'req-29c39ad8-e040-4d83-985c-9a7e555aae8b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.870056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '10095a8153824264ae08945d9f273d16', 'request_id': 'req-29c39ad8-e040-4d83-985c-9a7e555aae8b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.894242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] The token cache is being invalidate because project 10095a8153824264ae08945d9f273d16 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:48.894242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.894830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-29c39ad8-e040-4d83-985c-9a7e555aae8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:48.896866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 955/1904] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:48 2022] DELETE /identity/v3/projects/10095a8153824264ae08945d9f273d16 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:53.266075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:53.267626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:53.284383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8c68f16-deae-4ad5-9153-4bf5cf3d64f0 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:53.343593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:53.344321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:53.344609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:53.344908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] PATH_INFO: `/v3/users/52230ba322754f16b37c8fdc1b6ac119` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:53.346175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] RBAC: Authorizing `identity:delete_user(user_id=52230ba322754f16b37c8fdc1b6ac119)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:53.349335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:53.373989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:53.402542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52230ba322754f16b37c8fdc1b6ac119'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:53.412159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52230ba322754f16b37c8fdc1b6ac119'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:53.412427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52230ba322754f16b37c8fdc1b6ac119'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:53.412987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52230ba322754f16b37c8fdc1b6ac119'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:53.422544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52230ba322754f16b37c8fdc1b6ac119'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:53.445366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 950/1905] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:53 2022] DELETE /identity/v3/users/52230ba322754f16b37c8fdc1b6ac119 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:53.447510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:53.448011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:53.448100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:53.467089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:53.468631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:53.552448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 956/1906] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:53.566853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:53.568519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:53.637172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72969c79-6eca-4017-a933-3eaa652e0398 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:53.640854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:53.641471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:53.641741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:53.642011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:53.651609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:53.653234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:53.664060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 951/1907] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:53.937248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:53.939564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:53.952125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2dbbb90-d05d-474e-8f59-92ab172cb3d4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:53.958824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:53.959482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:53.959758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:53.960025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] PATH_INFO: `/v3/projects/24f36f040e3e4cd6bbc0cf0703a4cba3` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:53.962751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] RBAC: Authorizing `identity:delete_project(project_id=24f36f040e3e4cd6bbc0cf0703a4cba3)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:53.964700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:54.023293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:54.027748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24f36f040e3e4cd6bbc0cf0703a4cba3', 'request_id': 'req-58f40bfe-7d23-46f5-a909-6c4689782382'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.039323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24f36f040e3e4cd6bbc0cf0703a4cba3', 'request_id': 'req-58f40bfe-7d23-46f5-a909-6c4689782382'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.047329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24f36f040e3e4cd6bbc0cf0703a4cba3', 'request_id': 'req-58f40bfe-7d23-46f5-a909-6c4689782382'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.058596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] The token cache is being invalidate because project 24f36f040e3e4cd6bbc0cf0703a4cba3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:54.058879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.059564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.061232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 957/1908] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:53 2022] DELETE /identity/v3/projects/24f36f040e3e4cd6bbc0cf0703a4cba3 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:54.065522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:54.075107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcbf903-f072-4527-8fae-53fc857ae709 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:54.139439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:54.140078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:54.140353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:54.140620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] PATH_INFO: `/v3/users/84c7a324ef354f2b86d058dce9e5cf01` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:54.141847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] RBAC: Authorizing `identity:delete_user(user_id=84c7a324ef354f2b86d058dce9e5cf01)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:54.143819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:54.201836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:54.204392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '84c7a324ef354f2b86d058dce9e5cf01'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.214894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '84c7a324ef354f2b86d058dce9e5cf01'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.215911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '84c7a324ef354f2b86d058dce9e5cf01'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.217684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '84c7a324ef354f2b86d058dce9e5cf01'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.230778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c800eb27-74c0-49bd-aa2e-36db71a2e350 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '84c7a324ef354f2b86d058dce9e5cf01'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:54.253450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 952/1909] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:54 2022] DELETE /identity/v3/users/84c7a324ef354f2b86d058dce9e5cf01 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:55.057649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:55.059290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:55.073515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58f40bfe-7d23-46f5-a909-6c4689782382 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:55.146773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:55.147493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.147822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.148090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] PATH_INFO: `/v3/projects/8fe4fadd97494b8eb82e1353fec6dcd9` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.149342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fe4fadd97494b8eb82e1353fec6dcd9)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:55.151196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:55.239075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fe4fadd97494b8eb82e1353fec6dcd9', 'request_id': 'req-f9699cfa-59db-4804-8f07-30d94283f715'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:55.248703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fe4fadd97494b8eb82e1353fec6dcd9', 'request_id': 'req-f9699cfa-59db-4804-8f07-30d94283f715'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:55.258682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fe4fadd97494b8eb82e1353fec6dcd9', 'request_id': 'req-f9699cfa-59db-4804-8f07-30d94283f715'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:55.272531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] The token cache is being invalidate because project 8fe4fadd97494b8eb82e1353fec6dcd9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:55.272911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:55.273543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f9699cfa-59db-4804-8f07-30d94283f715 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:55.275487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 958/1910] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:55 2022] DELETE /identity/v3/projects/8fe4fadd97494b8eb82e1353fec6dcd9 => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:55.280169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efd61a3-cb0f-4fae-9bbe-054567a4450c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.281655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efd61a3-cb0f-4fae-9bbe-054567a4450c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.282164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1efd61a3-cb0f-4fae-9bbe-054567a4450c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.308148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1efd61a3-cb0f-4fae-9bbe-054567a4450c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:55.309379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1efd61a3-cb0f-4fae-9bbe-054567a4450c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:55.388909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 953/1911] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:55.397658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f202ce-331e-4ed5-9888-d0f76a4edf79 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.397946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f202ce-331e-4ed5-9888-d0f76a4edf79 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.398247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64f202ce-331e-4ed5-9888-d0f76a4edf79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.423694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-64f202ce-331e-4ed5-9888-d0f76a4edf79 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:55.424924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-64f202ce-331e-4ed5-9888-d0f76a4edf79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:55.432960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 959/1912] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:55.437324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efd61a3-cb0f-4fae-9bbe-054567a4450c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:55.447607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1efd61a3-cb0f-4fae-9bbe-054567a4450c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:55.452186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:55.452804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.453074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.453383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.454072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:55.454376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:55.456021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:55.468575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 954/1913] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:13:55.473251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f202ce-331e-4ed5-9888-d0f76a4edf79 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:55.483178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64f202ce-331e-4ed5-9888-d0f76a4edf79 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:55.488721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:55.489357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.489625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.489882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.490542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:55.492447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:55.524174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 960/1914] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:55 2022] POST /identity/v3/projects => generated 367 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:55.525281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:55.535558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51cc3bfc-c396-49c1-a862-768bec9d9066 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:55.605409 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:55.606051 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.606333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.606601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.607244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:55.610033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:55.633281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 955/1915] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:55 2022] POST /identity/v3/users => generated 284 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:55.637447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:55.647922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c97ed3-412d-4698-88fd-88edd08ed079 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:55.652399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:55.653456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.653456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.653644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.654178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:55.655775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:55.671949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 961/1916] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:55 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:55.677649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:55.691419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ecd6d2-6fd3-4aba-aeee-5324ba513e36 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:55.696687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe7afd9-cfee-4c63-870b-c787bc9910cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:55.697288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe7afd9-cfee-4c63-870b-c787bc9910cd admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.697560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe7afd9-cfee-4c63-870b-c787bc9910cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.697824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe7afd9-cfee-4c63-870b-c787bc9910cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.698527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe7afd9-cfee-4c63-870b-c787bc9910cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:55.700134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fe7afd9-cfee-4c63-870b-c787bc9910cd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:55.711544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 956/1917] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:55 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:55.715935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:55.726841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7742b6-fd8e-45dd-b6e9-c14aa4649c4f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:55.731531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:55.732197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.732469 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.732801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] PATH_INFO: `/v3/projects/b08cc9b9340d42d3be0dcafa2889d40b/users/ee036fd0c6c7400eaa4d93ca14e655cb/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.752521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b08cc9b9340d42d3be0dcafa2889d40b, user_id=ee036fd0c6c7400eaa4d93ca14e655cb, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:55.756596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:55.768835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:55.777460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 962/1918] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:55 2022] PUT /identity/v3/projects/b08cc9b9340d42d3be0dcafa2889d40b/users/ee036fd0c6c7400eaa4d93ca14e655cb/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:55.794038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06401c8b-1172-4b18-a343-74eaf3008cd6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:55.794420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06401c8b-1172-4b18-a343-74eaf3008cd6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:55.794931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06401c8b-1172-4b18-a343-74eaf3008cd6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:55.837242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-06401c8b-1172-4b18-a343-74eaf3008cd6 None None] MFA Rules not processed for user `ee036fd0c6c7400eaa4d93ca14e655cb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:55.839239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-06401c8b-1172-4b18-a343-74eaf3008cd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:55.855748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:55.967796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 957/1919] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:55 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:55.968869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.129089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8257998-d3fc-4833-b856-63d847e11ac9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.141220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.141889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.142214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.142486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.146960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:56.153894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:56.217336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06401c8b-1172-4b18-a343-74eaf3008cd6 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.217838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 963/1920] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:55 2022] POST /identity/v3/projects => generated 367 bytes in 245 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:56.232855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06401c8b-1172-4b18-a343-74eaf3008cd6 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.247959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.249868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:56.356959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.357883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.358130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.358544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.359243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:56.372577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:56.396618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5d1f22-6222-4499-9ef7-70e71fd36798 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.404408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.405558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.406450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.408481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] PATH_INFO: `/v3/users/ac7c610a831a41e59614a9aa2b703b89` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.410395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac7c610a831a41e59614a9aa2b703b89)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:56.413398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:56.422881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 958/1921] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:56 2022] POST /identity/v3/users => generated 290 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:56.425773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.461603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4237a8-d9b1-4adb-af92-0454a7fb1e0f admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.467774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.468939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.469745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.470182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.470978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:56.472753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:56.498439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 959/1922] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:56 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:56.502825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.527350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae6824bd-4d45-41d2-8a1b-b70d728a91c6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.528081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac7c610a831a41e59614a9aa2b703b89'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:56.553675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.554356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac7c610a831a41e59614a9aa2b703b89'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:56.554581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.554961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.556344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] PATH_INFO: `/v3/projects/71a93a15b9c9439e80e58afceade98be/users/47d104142c7746e7bc76e6c1d4098b4f/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.557140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac7c610a831a41e59614a9aa2b703b89'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:56.557957 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac7c610a831a41e59614a9aa2b703b89'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:56.584083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-97e2086b-2b69-4823-b5ee-36653623f069 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac7c610a831a41e59614a9aa2b703b89'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:56.596436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71a93a15b9c9439e80e58afceade98be, user_id=47d104142c7746e7bc76e6c1d4098b4f, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:56.598867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:56.624261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 964/1923] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:56 2022] DELETE /identity/v3/users/ac7c610a831a41e59614a9aa2b703b89 => generated 0 bytes in 372 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:56.626352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.626857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.627504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.635356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 960/1924] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:56 2022] PUT /identity/v3/projects/71a93a15b9c9439e80e58afceade98be/users/47d104142c7746e7bc76e6c1d4098b4f/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:56.636536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.639173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:56.640363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:56.664853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:56.669045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:56.774777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2c8a2d1-cc9d-4e4a-9ecd-344aac2690c7 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.783553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.784183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.784466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.784727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.785471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:56.787082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:56.797883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 961/1925] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:56 2022] GET /identity/v3/roles => generated 1420 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:13:56.802058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.811313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 965/1926] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:56.812464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65ee9a34-ee64-405a-b133-4da5e5280e84 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.817423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651f3b15-36c3-46ed-8171-97ba5cd1dd15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.818194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f3b15-36c3-46ed-8171-97ba5cd1dd15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.818536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f3b15-36c3-46ed-8171-97ba5cd1dd15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.818871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651f3b15-36c3-46ed-8171-97ba5cd1dd15 admin admin] PATH_INFO: `/v3/projects/71a93a15b9c9439e80e58afceade98be/users/47d104142c7746e7bc76e6c1d4098b4f/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.821295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651f3b15-36c3-46ed-8171-97ba5cd1dd15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=71a93a15b9c9439e80e58afceade98be, user_id=47d104142c7746e7bc76e6c1d4098b4f, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:56.822207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.824051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:56.824179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651f3b15-36c3-46ed-8171-97ba5cd1dd15 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:56.826676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:56.846145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 962/1927] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:56 2022] PUT /identity/v3/projects/71a93a15b9c9439e80e58afceade98be/users/47d104142c7746e7bc76e6c1d4098b4f/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:13:56.849476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.849943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.850247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.879273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] MFA Rules not processed for user `47d104142c7746e7bc76e6c1d4098b4f`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:56.880483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:56.889759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-624a2b3e-5051-489d-a6af-ea0b2f38b05f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:56.942217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:56.942762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:56.943035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:56.943351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:56.962279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 963/1928] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:56 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:13:56.976822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:56.979067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:57.015500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:57.017131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:57.027032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 966/1929] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 205 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:57.040655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7185058-132b-4375-92b6-d681507abbb2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:57.044454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:57.045187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:57.045566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:57.045935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:57.057304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:57.059312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:57.074186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 964/1930] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:56 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:13:57.171945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:57.181935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe2e6ab1-18e5-4c70-83f4-873250cb0b18 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:57.186506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:57.187258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:57.187583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:57.188227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:57.189883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:57.197390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:57.269696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:57.271193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:57.280387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 967/1931] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:57 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:57.345252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:57.346831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:57.362137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-adbb423c-bcde-4ceb-a929-c023316c684a service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:57.376824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:57.377487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:57.377767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:57.378176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] PATH_INFO: `/v3/projects/65dafa5bd1fa4ada875c3102789d6b30` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:57.379195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=65dafa5bd1fa4ada875c3102789d6b30)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:57.381207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:57.443481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '65dafa5bd1fa4ada875c3102789d6b30', 'request_id': 'req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.457301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '65dafa5bd1fa4ada875c3102789d6b30', 'request_id': 'req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.467762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '65dafa5bd1fa4ada875c3102789d6b30', 'request_id': 'req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.481980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] The token cache is being invalidate because project 65dafa5bd1fa4ada875c3102789d6b30 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:57.482403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.483167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.485124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 965/1932] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:57 2022] DELETE /identity/v3/projects/65dafa5bd1fa4ada875c3102789d6b30 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:57.489621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:57.508395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9348e46f-c1ed-4a13-b77a-02a5c9aa1523 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:57.645510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:57.646302 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:57.646584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:57.646858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] PATH_INFO: `/v3/users/a195a80db0a94235bc34ec1bc06f0d8a` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:57.648932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a195a80db0a94235bc34ec1bc06f0d8a)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:57.650947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:57.759904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a195a80db0a94235bc34ec1bc06f0d8a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.769251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a195a80db0a94235bc34ec1bc06f0d8a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.770121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a195a80db0a94235bc34ec1bc06f0d8a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.770824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a195a80db0a94235bc34ec1bc06f0d8a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.785205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c9d7612-db76-4b1a-972a-548f7a1099e2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a195a80db0a94235bc34ec1bc06f0d8a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:57.820822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 968/1933] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:13:57 2022] DELETE /identity/v3/users/a195a80db0a94235bc34ec1bc06f0d8a => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:58.116983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.118611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:58.134680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:58.138684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1389a6-cfae-47a7-be9d-4dff5a4cebb1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.202157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.203208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.203471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.203854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] PATH_INFO: `/v3/projects/0a2b3767ba564071bef109e40a567777` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.205107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a2b3767ba564071bef109e40a567777)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.207073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.256362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a2b3767ba564071bef109e40a567777', 'request_id': 'req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:58.267709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a2b3767ba564071bef109e40a567777', 'request_id': 'req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:58.275316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a2b3767ba564071bef109e40a567777', 'request_id': 'req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:58.287191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] The token cache is being invalidate because project 0a2b3767ba564071bef109e40a567777 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:13:58.287475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:58.288150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-5b6f5a34-c2ad-4a5d-8a20-e613a8152f0a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:13:58.289655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 966/1934] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:13:58 2022] DELETE /identity/v3/projects/0a2b3767ba564071bef109e40a567777 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:58.294665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d370612f-980a-4f5d-94c4-d36c5184efc0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.295191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d370612f-980a-4f5d-94c4-d36c5184efc0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.295191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d370612f-980a-4f5d-94c4-d36c5184efc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.315615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d370612f-980a-4f5d-94c4-d36c5184efc0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:58.316895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d370612f-980a-4f5d-94c4-d36c5184efc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:58.338626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:58.379348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 969/1935] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:58.387371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d11a62c-6b2b-4dc7-a311-dcd13934e4f5 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.387884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d11a62c-6b2b-4dc7-a311-dcd13934e4f5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.387884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d11a62c-6b2b-4dc7-a311-dcd13934e4f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.408133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7d11a62c-6b2b-4dc7-a311-dcd13934e4f5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:58.409346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7d11a62c-6b2b-4dc7-a311-dcd13934e4f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:58.417503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 967/1936] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:58.421941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d370612f-980a-4f5d-94c4-d36c5184efc0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.438123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d370612f-980a-4f5d-94c4-d36c5184efc0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.443218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.443807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.444082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.444348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.444999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:13:58.445365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.447538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.463860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 970/1937] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:13:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:13:58.469835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d11a62c-6b2b-4dc7-a311-dcd13934e4f5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.480186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d11a62c-6b2b-4dc7-a311-dcd13934e4f5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.485807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.486446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.486725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.486993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.487638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.489566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.519833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 968/1938] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:58 2022] POST /identity/v3/projects => generated 371 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:58.524032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.534996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d34cc3d-2478-4dab-b0a7-e04ee1c5f0fe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.606732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.607399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.607676 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.607969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.608616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.611056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.635648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 971/1939] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:58 2022] POST /identity/v3/users => generated 286 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:13:58.640025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.650653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52984af2-7a38-4105-8767-fa4fc78cb18d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.656060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.656725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.657005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.657276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.657917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.659565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.670299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 969/1940] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:58.675064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.685798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-336eb91e-edca-4e70-a00d-53dfddb8326b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.691154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9541bbcc-1391-4bd1-b692-082b597d2c40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.691798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9541bbcc-1391-4bd1-b692-082b597d2c40 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.692062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9541bbcc-1391-4bd1-b692-082b597d2c40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.692335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9541bbcc-1391-4bd1-b692-082b597d2c40 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.692998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9541bbcc-1391-4bd1-b692-082b597d2c40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.694656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9541bbcc-1391-4bd1-b692-082b597d2c40 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.705833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 972/1941] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:13:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:13:58.710389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.721186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42cbb428-23ad-4e87-8e23-ac6f13b9bea7 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.726483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.727145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.727421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.727691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] PATH_INFO: `/v3/projects/d39411cc4b0a4d00997bebdef0c210b7/users/f49c6293f9594461903218002a9386b9/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.742809 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] RBAC: Authorizing `identity:create_grant(project_id=d39411cc4b0a4d00997bebdef0c210b7, user_id=f49c6293f9594461903218002a9386b9, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.745480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.760643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 970/1942] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:13:58 2022] PUT /identity/v3/projects/d39411cc4b0a4d00997bebdef0c210b7/users/f49c6293f9594461903218002a9386b9/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:13:58.766134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93490f-d817-4665-9006-2b01bf2ce590 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.767057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93490f-d817-4665-9006-2b01bf2ce590 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.767341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e93490f-d817-4665-9006-2b01bf2ce590 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.794181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0e93490f-d817-4665-9006-2b01bf2ce590 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:58.794181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0e93490f-d817-4665-9006-2b01bf2ce590 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:58.863100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 973/1943] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:58.869136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:58.871041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:58.908966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:58.935996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b297c8-90e1-4716-bcb2-145c1f5973dd admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:58.941066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:58.941790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:58.942094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:58.942364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:58.954516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:58.955950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:58.966893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 971/1944] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:13:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:13:59.764704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d5a73-7428-4c1d-a691-6434968acbbd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:59.765315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d5a73-7428-4c1d-a691-6434968acbbd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:59.765315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd1d5a73-7428-4c1d-a691-6434968acbbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:59.794627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fd1d5a73-7428-4c1d-a691-6434968acbbd None None] MFA Rules not processed for user `f49c6293f9594461903218002a9386b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:13:59.795922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fd1d5a73-7428-4c1d-a691-6434968acbbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:13:59.821920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:13:59.880219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 974/1945] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:13:59 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:13:59.885158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:59.895657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e68eb3e-08c6-4a06-905b-96ff4ecc785c service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:13:59.906727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:13:59.908069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:13:59.908978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:13:59.909421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:13:59.910269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:13:59.912245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:13:59.950350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 972/1946] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:13:59 2022] POST /identity/v3/projects => generated 371 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:13:59.952319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d5a73-7428-4c1d-a691-6434968acbbd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:13:59.965117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd1d5a73-7428-4c1d-a691-6434968acbbd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:00.026346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:00.026963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:00.027637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:00.027911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:00.028872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:00.030464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:00.055071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 975/1947] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:13:59 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:00.059015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:00.069650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-878c6dd6-fe55-4782-8e3e-413cf1301ed4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:00.074489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:00.075013 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:00.075473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:00.075785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:00.076412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:00.078026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:00.090599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 973/1948] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:00.096003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:00.106640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea033b4c-28a6-4683-b26e-899fe7b445a1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:00.117548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:00.118278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:00.118553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:00.118820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] PATH_INFO: `/v3/projects/7f2a1bc7984e42d39fa9fd78e3763711/users/72efe7d3b30e48fa94793ec712bb9370/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:00.136101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f2a1bc7984e42d39fa9fd78e3763711, user_id=72efe7d3b30e48fa94793ec712bb9370, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:00.138808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:00.154359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 976/1949] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:00 2022] PUT /identity/v3/projects/7f2a1bc7984e42d39fa9fd78e3763711/users/72efe7d3b30e48fa94793ec712bb9370/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:00.158816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:00.169437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231363b6-767a-4151-bda6-c731d47eab89 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:00.229022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04334662-d47b-4b3d-ba22-29aa06d8e51e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:00.229637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04334662-d47b-4b3d-ba22-29aa06d8e51e admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:00.229912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04334662-d47b-4b3d-ba22-29aa06d8e51e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:00.230474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04334662-d47b-4b3d-ba22-29aa06d8e51e admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:00.231114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04334662-d47b-4b3d-ba22-29aa06d8e51e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:00.233457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04334662-d47b-4b3d-ba22-29aa06d8e51e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:00.249127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:00.251904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 974/1950] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:00 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:00.261272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eec3fe5-0bc3-47a3-93dd-addf7ccf6606 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:00.267518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:00.268293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:00.268637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:00.268913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] PATH_INFO: `/v3/projects/7f2a1bc7984e42d39fa9fd78e3763711/users/72efe7d3b30e48fa94793ec712bb9370/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:00.270917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7f2a1bc7984e42d39fa9fd78e3763711, user_id=72efe7d3b30e48fa94793ec712bb9370, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:00.273634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:00.286278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 977/1951] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:00 2022] PUT /identity/v3/projects/7f2a1bc7984e42d39fa9fd78e3763711/users/72efe7d3b30e48fa94793ec712bb9370/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:01.005141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:01.005679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:01.005679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:01.034842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] MFA Rules not processed for user `72efe7d3b30e48fa94793ec712bb9370`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:01.037099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:01.126978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 975/1952] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:01 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:01.140727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:01.142856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:01.242622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c72e18-8e42-48b5-a84e-8a52c91db008 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:01.249115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6aca05-0a60-400f-8973-9bd1f465a857 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:01.249735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6aca05-0a60-400f-8973-9bd1f465a857 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:01.250029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6aca05-0a60-400f-8973-9bd1f465a857 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:01.250303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6aca05-0a60-400f-8973-9bd1f465a857 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:01.283014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:01.314982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6aca05-0a60-400f-8973-9bd1f465a857 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:01.316420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6aca05-0a60-400f-8973-9bd1f465a857 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:01.325358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 978/1953] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:01 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:02.384502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:02.386230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:02.414225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:02.455433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-402971ab-1612-46bc-8882-8ec4c21b288a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:02.460304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:02.460875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:02.461148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:02.461415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] PATH_INFO: `/v3/users/ee036fd0c6c7400eaa4d93ca14e655cb` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:02.462712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ee036fd0c6c7400eaa4d93ca14e655cb)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:02.464666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:02.519295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ee036fd0c6c7400eaa4d93ca14e655cb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:02.526023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ee036fd0c6c7400eaa4d93ca14e655cb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:02.526679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ee036fd0c6c7400eaa4d93ca14e655cb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:02.527271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ee036fd0c6c7400eaa4d93ca14e655cb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:02.537167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ee036fd0c6c7400eaa4d93ca14e655cb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:02.555411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 976/1954] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:02 2022] DELETE /identity/v3/users/ee036fd0c6c7400eaa4d93ca14e655cb => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:02.560367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:02.560830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:02.560940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:02.581106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:02.582382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:02.646516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 979/1955] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:02.656553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:02.658336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:02.717721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3229d1bc-3385-4577-afd7-0a018cc56180 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:02.727637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:02.728313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:02.728591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:02.728875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:02.741907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:02.743987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:02.755933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 977/1956] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:02.795994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:02.797685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:02.884708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-250cdd88-c7f7-4f4a-9d51-dea93fcd87be None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:02.889485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:02.890721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:02.891433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:02.892037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:02.896474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:02.998258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:03.000331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:03.022877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 980/1957] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:02 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 228 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:03.231964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:03.234196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:03.275784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa52308c-fa8e-4729-9e49-5d83af1dcd47 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:03.281979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:03.283142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:03.283355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:03.283625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] PATH_INFO: `/v3/projects/b08cc9b9340d42d3be0dcafa2889d40b` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:03.285526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b08cc9b9340d42d3be0dcafa2889d40b)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:03.288147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:03.352781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:03.367219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b08cc9b9340d42d3be0dcafa2889d40b', 'request_id': 'req-abced70a-18ae-4b14-8972-139b02bfc68e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.389355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b08cc9b9340d42d3be0dcafa2889d40b', 'request_id': 'req-abced70a-18ae-4b14-8972-139b02bfc68e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.397739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b08cc9b9340d42d3be0dcafa2889d40b', 'request_id': 'req-abced70a-18ae-4b14-8972-139b02bfc68e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.414487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] The token cache is being invalidate because project b08cc9b9340d42d3be0dcafa2889d40b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:03.414756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.415425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.417478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 978/1958] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:03 2022] DELETE /identity/v3/projects/b08cc9b9340d42d3be0dcafa2889d40b => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:03.421732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:03.445879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2786c5-2663-4cbd-b550-9b3e1bd4d874 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:03.465943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:03.519085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:03.520235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:03.520744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:03.522412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] PATH_INFO: `/v3/users/47d104142c7746e7bc76e6c1d4098b4f` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:03.523736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47d104142c7746e7bc76e6c1d4098b4f)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:03.526335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:03.607906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47d104142c7746e7bc76e6c1d4098b4f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.618967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47d104142c7746e7bc76e6c1d4098b4f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.619723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47d104142c7746e7bc76e6c1d4098b4f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.620304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47d104142c7746e7bc76e6c1d4098b4f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.632206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-bfccc6f0-9347-49ef-829f-6b67e19015c7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47d104142c7746e7bc76e6c1d4098b4f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:03.668666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 981/1959] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:03 2022] DELETE /identity/v3/users/47d104142c7746e7bc76e6c1d4098b4f => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:03.997137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:03.998958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:04.014068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abced70a-18ae-4b14-8972-139b02bfc68e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.086680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.087405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.088244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.088521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] PATH_INFO: `/v3/projects/71a93a15b9c9439e80e58afceade98be` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.089640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] RBAC: Authorizing `identity:delete_project(project_id=71a93a15b9c9439e80e58afceade98be)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.091624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:04.145356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '71a93a15b9c9439e80e58afceade98be', 'request_id': 'req-eda8d6d2-a1f5-4344-bdfa-774059159e70'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:04.155463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '71a93a15b9c9439e80e58afceade98be', 'request_id': 'req-eda8d6d2-a1f5-4344-bdfa-774059159e70'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:04.162674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '71a93a15b9c9439e80e58afceade98be', 'request_id': 'req-eda8d6d2-a1f5-4344-bdfa-774059159e70'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:04.176414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] The token cache is being invalidate because project 71a93a15b9c9439e80e58afceade98be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:04.176414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:04.177018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-eda8d6d2-a1f5-4344-bdfa-774059159e70 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:04.184158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3d5fd3-780f-47cc-9df0-174f63a2d8b4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.184686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3d5fd3-780f-47cc-9df0-174f63a2d8b4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.185060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f3d5fd3-780f-47cc-9df0-174f63a2d8b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.186081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 979/1960] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:03 2022] DELETE /identity/v3/projects/71a93a15b9c9439e80e58afceade98be => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:04.209992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2f3d5fd3-780f-47cc-9df0-174f63a2d8b4 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:04.211622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2f3d5fd3-780f-47cc-9df0-174f63a2d8b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:04.291270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 982/1961] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:04.300497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f60c78-d67b-46dd-916a-77cb4639ed68 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.300830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f60c78-d67b-46dd-916a-77cb4639ed68 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.301098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84f60c78-d67b-46dd-916a-77cb4639ed68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.321327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-84f60c78-d67b-46dd-916a-77cb4639ed68 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:04.322577 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-84f60c78-d67b-46dd-916a-77cb4639ed68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:04.330502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 980/1962] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:04.335140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3d5fd3-780f-47cc-9df0-174f63a2d8b4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:04.345592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f3d5fd3-780f-47cc-9df0-174f63a2d8b4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.350912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.351618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.351957 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.352471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.353178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:04.353541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.355247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:04.367853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 983/1963] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:14:04.372280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f60c78-d67b-46dd-916a-77cb4639ed68 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:04.382127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84f60c78-d67b-46dd-916a-77cb4639ed68 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.387596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.388073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.388358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.389009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.389309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.391357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:04.424795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 981/1964] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:04 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:04.426090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:04.436893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27ffa9ce-fd41-43eb-87fd-a9a65801f845 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.463397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:04.518420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.519483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.520053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.520448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.521234 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.523327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:04.560273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 984/1965] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:04 2022] POST /identity/v3/users => generated 292 bytes in 135 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:04.564446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:04.578398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a812c9c2-4fbe-4933-b4da-ef0333e284b0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.585052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.586188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.586562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.588728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.589648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.592155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:04.616836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 982/1966] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:04 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:04.617734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:04.630205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38d04d80-8f8d-495d-9ac1-313c18faa774 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.637375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92ccf18-fe7e-4190-97eb-724cd360c284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.638075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ccf18-fe7e-4190-97eb-724cd360c284 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.638690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ccf18-fe7e-4190-97eb-724cd360c284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.638940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92ccf18-fe7e-4190-97eb-724cd360c284 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.639595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92ccf18-fe7e-4190-97eb-724cd360c284 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.641193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c92ccf18-fe7e-4190-97eb-724cd360c284 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:04.662250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 985/1967] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:04 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:04.663842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:04.693757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2082cbd1-25a1-4045-85dd-bca1433b5e83 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.699508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.700381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.700772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.701155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] PATH_INFO: `/v3/projects/e81db9db49f1440c81bcde8900bf1aa1/users/ad4aa3cf305e4b63b40e6a2c66281b9d/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.717504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] RBAC: Authorizing `identity:create_grant(project_id=e81db9db49f1440c81bcde8900bf1aa1, user_id=ad4aa3cf305e4b63b40e6a2c66281b9d, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.720867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:04.723632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:04.748754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 983/1968] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:04 2022] PUT /identity/v3/projects/e81db9db49f1440c81bcde8900bf1aa1/users/ad4aa3cf305e4b63b40e6a2c66281b9d/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:04.758610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fe1050-9998-4553-b183-fa57060b61ad None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.758784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fe1050-9998-4553-b183-fa57060b61ad None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.759110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fe1050-9998-4553-b183-fa57060b61ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.800244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-88fe1050-9998-4553-b183-fa57060b61ad None None] MFA Rules not processed for user `ad4aa3cf305e4b63b40e6a2c66281b9d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:04.803907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-88fe1050-9998-4553-b183-fa57060b61ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:04.915651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 986/1969] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:04 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:04.920299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:04.986186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b69cf3f5-9e78-4bd5-b914-ae883a2d35fd admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:04.991014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:04.991660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:04.991931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:04.992198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:04.992827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:04.994829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.025930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 984/1970] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:04 2022] POST /identity/v3/projects => generated 385 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:05.030173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fe1050-9998-4553-b183-fa57060b61ad None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.042680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fe1050-9998-4553-b183-fa57060b61ad None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.105067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.105715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.106027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.106449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.107133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.109065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.137834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 987/1971] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:05 2022] POST /identity/v3/users => generated 293 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:05.139230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.153351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7cec62-1ec5-4295-b5c8-fd43c156be49 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.162096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.163112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.163402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.163720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.164524 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.166293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.185612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 985/1972] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:05 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:05.187601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.199328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c3e29ae-14a3-410a-b3c9-0e6cfdb64388 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.205920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee33f17f-c735-4cf3-bc86-9abe2796567d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.206657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee33f17f-c735-4cf3-bc86-9abe2796567d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.206977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee33f17f-c735-4cf3-bc86-9abe2796567d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.207322 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee33f17f-c735-4cf3-bc86-9abe2796567d admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.208135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee33f17f-c735-4cf3-bc86-9abe2796567d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.210071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee33f17f-c735-4cf3-bc86-9abe2796567d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.226294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 988/1973] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:05 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:05.227981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.241850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7ee8a08-1567-4ac7-a5f4-43625978bdd2 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.259368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.260355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.261382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.261842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] PATH_INFO: `/v3/projects/f0aa03be425c44528d9e254f56df80c4/users/5a90b9d426394962a129e694fa3c8994/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.284493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f0aa03be425c44528d9e254f56df80c4, user_id=5a90b9d426394962a129e694fa3c8994, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.288625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.332282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 986/1974] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:05 2022] PUT /identity/v3/projects/f0aa03be425c44528d9e254f56df80c4/users/5a90b9d426394962a129e694fa3c8994/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:05.343033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7a262d-3480-41ec-aa6b-aaca2dea7242 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.343317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7a262d-3480-41ec-aa6b-aaca2dea7242 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.344598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f7a262d-3480-41ec-aa6b-aaca2dea7242 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.372664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:05.378301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3f7a262d-3480-41ec-aa6b-aaca2dea7242 None None] MFA Rules not processed for user `5a90b9d426394962a129e694fa3c8994`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:05.379770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3f7a262d-3480-41ec-aa6b-aaca2dea7242 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:05.463070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 989/1975] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:05 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:05.467603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.559491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9520565-407c-42f1-a1b6-029414dabc3c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.564981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.565629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.565903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.566204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.566857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.569228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.600528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 987/1976] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:05 2022] POST /identity/v3/projects => generated 385 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:05.604769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7a262d-3480-41ec-aa6b-aaca2dea7242 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.616324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f7a262d-3480-41ec-aa6b-aaca2dea7242 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.684485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.685139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.685427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.686579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.686579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.688460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.720413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 990/1977] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:05 2022] POST /identity/v3/users => generated 299 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:05.725901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.740044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcefa0cb-5124-4f4f-a9cf-3e7adc92b714 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.744639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.745244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.745517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.745782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.746461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.748056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.766063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 988/1978] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:05.768387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.783511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fc6b0c1-b42d-4751-957b-bcb9167ae6c8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.788274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.788944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.789280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.789559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] PATH_INFO: `/v3/projects/aaf204d854ab447d8a46c5396e0abe43/users/ca9647c8a98a4b3a8c455860fa86eb0d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.807758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaf204d854ab447d8a46c5396e0abe43, user_id=ca9647c8a98a4b3a8c455860fa86eb0d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.811157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.832168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 991/1979] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:05 2022] PUT /identity/v3/projects/aaf204d854ab447d8a46c5396e0abe43/users/ca9647c8a98a4b3a8c455860fa86eb0d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:05.836054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.848326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564542f4-70df-44c3-b388-4c019b930e38 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.864195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:05.932768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5de1d598-ff4f-4eec-8e46-b8f65ba1187f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.933534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1d598-ff4f-4eec-8e46-b8f65ba1187f admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.933859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1d598-ff4f-4eec-8e46-b8f65ba1187f admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.934164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5de1d598-ff4f-4eec-8e46-b8f65ba1187f admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.935174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1d598-ff4f-4eec-8e46-b8f65ba1187f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.936436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5de1d598-ff4f-4eec-8e46-b8f65ba1187f admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.947011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 989/1980] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:05 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:05.951083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:05.961942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97bffe1d-2a53-4c78-9849-21e74b8d5815 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:05.969166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:05.970475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.970475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.970475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] PATH_INFO: `/v3/projects/aaf204d854ab447d8a46c5396e0abe43/users/ca9647c8a98a4b3a8c455860fa86eb0d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:05.972693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aaf204d854ab447d8a46c5396e0abe43, user_id=ca9647c8a98a4b3a8c455860fa86eb0d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:05.975250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:05.987323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 992/1981] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:05 2022] PUT /identity/v3/projects/aaf204d854ab447d8a46c5396e0abe43/users/ca9647c8a98a4b3a8c455860fa86eb0d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:05.999382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ddfddc-b293-4456-923f-c0b8583129e2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:05.999690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ddfddc-b293-4456-923f-c0b8583129e2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:05.999953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1ddfddc-b293-4456-923f-c0b8583129e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:06.032774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a1ddfddc-b293-4456-923f-c0b8583129e2 None None] MFA Rules not processed for user `ca9647c8a98a4b3a8c455860fa86eb0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:06.034760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a1ddfddc-b293-4456-923f-c0b8583129e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:06.119734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 990/1982] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:05 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:06.132337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:06.134111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:06.199650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87dda291-5924-4c92-95e0-49c0aeeda5f5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:06.216287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:06.216287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:06.216287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:06.216287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:06.216854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:06.268262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:06.269706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:06.278737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 993/1983] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:06 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:07.899181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ddfddc-b293-4456-923f-c0b8583129e2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:07.909470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1ddfddc-b293-4456-923f-c0b8583129e2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:07.913869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:07.914509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:07.914783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:07.915049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:07.974644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:07.976081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:07.985078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 991/1984] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:08.607686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:08.617166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b74af3e-d361-4006-86db-bdeef593c6ca service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:08.620760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4483b599-f37a-4188-8b91-0316fbc77d6d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:08.621355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4483b599-f37a-4188-8b91-0316fbc77d6d service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:08.621624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4483b599-f37a-4188-8b91-0316fbc77d6d service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:08.621888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4483b599-f37a-4188-8b91-0316fbc77d6d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:08.630992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4483b599-f37a-4188-8b91-0316fbc77d6d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:08.632481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4483b599-f37a-4188-8b91-0316fbc77d6d service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:08.642251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 994/1985] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:08 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:13.776147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:13.820483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:13.844595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d574712a-b6c8-4de8-b01e-10c874503ccb service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:13.849656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:13.850324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.850604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.850874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] PATH_INFO: `/v3/users/ad4aa3cf305e4b63b40e6a2c66281b9d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.851989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ad4aa3cf305e4b63b40e6a2c66281b9d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:13.854120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:13.911161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ad4aa3cf305e4b63b40e6a2c66281b9d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.919084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ad4aa3cf305e4b63b40e6a2c66281b9d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.920255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ad4aa3cf305e4b63b40e6a2c66281b9d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.922901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ad4aa3cf305e4b63b40e6a2c66281b9d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.942989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ad4aa3cf305e4b63b40e6a2c66281b9d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:13.966060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 992/1986] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:13 2022] DELETE /identity/v3/users/ad4aa3cf305e4b63b40e6a2c66281b9d => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:13.971108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:13.971770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:13.971770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:13.991495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:13.992697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:14.056357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 995/1987] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:14.066535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:14.067911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:14.126537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfaa80ed-9cee-453a-a1ec-d171b2031a48 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:14.130292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:14.130898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:14.131170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:14.131432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:14.140122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:14.141584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:14.150963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 993/1988] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:14.496353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:14.497853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:14.511121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e61491-fe24-4a66-ac9c-4d141c2f3aa1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:14.516253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:14.517233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:14.518016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:14.518280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] PATH_INFO: `/v3/projects/e81db9db49f1440c81bcde8900bf1aa1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:14.519478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e81db9db49f1440c81bcde8900bf1aa1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:14.521518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:14.570790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e81db9db49f1440c81bcde8900bf1aa1', 'request_id': 'req-e2f3a3e9-61b5-49f0-9346-852f8345c919'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.579831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e81db9db49f1440c81bcde8900bf1aa1', 'request_id': 'req-e2f3a3e9-61b5-49f0-9346-852f8345c919'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.586595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e81db9db49f1440c81bcde8900bf1aa1', 'request_id': 'req-e2f3a3e9-61b5-49f0-9346-852f8345c919'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.597713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] The token cache is being invalidate because project e81db9db49f1440c81bcde8900bf1aa1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:14.598088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.598666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.600164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 996/1989] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:14 2022] DELETE /identity/v3/projects/e81db9db49f1440c81bcde8900bf1aa1 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:14.604476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:14.614737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeee11aa-3421-443f-a27b-c1bc563eee33 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:14.687106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:14.688012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:14.688412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:14.688923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] PATH_INFO: `/v3/users/5a90b9d426394962a129e694fa3c8994` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:14.690266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a90b9d426394962a129e694fa3c8994)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:14.692268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:14.693846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:14.749263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a90b9d426394962a129e694fa3c8994'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.756825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a90b9d426394962a129e694fa3c8994'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.757517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a90b9d426394962a129e694fa3c8994'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.758253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a90b9d426394962a129e694fa3c8994'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.768855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a90b9d426394962a129e694fa3c8994'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:14.785968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 994/1990] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:14 2022] DELETE /identity/v3/users/5a90b9d426394962a129e694fa3c8994 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:15.079009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:15.080899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:15.096268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f3a3e9-61b5-49f0-9346-852f8345c919 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:15.117967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:15.163547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:15.164197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:15.164474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:15.164748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] PATH_INFO: `/v3/projects/f0aa03be425c44528d9e254f56df80c4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:15.165876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f0aa03be425c44528d9e254f56df80c4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:15.167822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:15.225111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f0aa03be425c44528d9e254f56df80c4', 'request_id': 'req-ffdaf533-fba8-493a-a46e-1a724f7ff758'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.235611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f0aa03be425c44528d9e254f56df80c4', 'request_id': 'req-ffdaf533-fba8-493a-a46e-1a724f7ff758'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.243091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f0aa03be425c44528d9e254f56df80c4', 'request_id': 'req-ffdaf533-fba8-493a-a46e-1a724f7ff758'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.260393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] The token cache is being invalidate because project f0aa03be425c44528d9e254f56df80c4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:15.260796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.261553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.263729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 997/1991] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:15 2022] DELETE /identity/v3/projects/f0aa03be425c44528d9e254f56df80c4 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:15.267624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:15.269678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:15.283999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ae659f-47df-446b-83e0-70d78dbee8d9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:15.353256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:15.353895 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:15.354189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:15.354459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] PATH_INFO: `/v3/users/ca9647c8a98a4b3a8c455860fa86eb0d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:15.355648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ca9647c8a98a4b3a8c455860fa86eb0d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:15.357583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:15.410376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ca9647c8a98a4b3a8c455860fa86eb0d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.418946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ca9647c8a98a4b3a8c455860fa86eb0d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.419597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ca9647c8a98a4b3a8c455860fa86eb0d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.420277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ca9647c8a98a4b3a8c455860fa86eb0d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.432280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8b9ff324-fb69-48ec-ba3e-d077694aa740 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ca9647c8a98a4b3a8c455860fa86eb0d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.449835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:15.457615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 995/1992] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:15 2022] DELETE /identity/v3/users/ca9647c8a98a4b3a8c455860fa86eb0d => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:15.796412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:15.798059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:15.811523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdaf533-fba8-493a-a46e-1a724f7ff758 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:15.871546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:15.872303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:15.872648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:15.872922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] PATH_INFO: `/v3/projects/aaf204d854ab447d8a46c5396e0abe43` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:15.874113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aaf204d854ab447d8a46c5396e0abe43)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:15.876217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:15.925362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aaf204d854ab447d8a46c5396e0abe43', 'request_id': 'req-8342017c-1aca-45f5-a6a3-e9d69550bea3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.927784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:15.933796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aaf204d854ab447d8a46c5396e0abe43', 'request_id': 'req-8342017c-1aca-45f5-a6a3-e9d69550bea3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.940989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aaf204d854ab447d8a46c5396e0abe43', 'request_id': 'req-8342017c-1aca-45f5-a6a3-e9d69550bea3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.952072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] The token cache is being invalidate because project aaf204d854ab447d8a46c5396e0abe43 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:15.952408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.953061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-8342017c-1aca-45f5-a6a3-e9d69550bea3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:15.954612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 998/1993] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:15 2022] DELETE /identity/v3/projects/aaf204d854ab447d8a46c5396e0abe43 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:15.959680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f838d1f-4b7b-4f87-8a4e-dee0b5e5f8d7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:15.960175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f838d1f-4b7b-4f87-8a4e-dee0b5e5f8d7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:15.960280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f838d1f-4b7b-4f87-8a4e-dee0b5e5f8d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:15.982546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0f838d1f-4b7b-4f87-8a4e-dee0b5e5f8d7 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:15.983763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0f838d1f-4b7b-4f87-8a4e-dee0b5e5f8d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:16.047434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 996/1994] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:16.056288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0fe46a-561a-4d95-933b-5b1cab6b01f5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.056571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0fe46a-561a-4d95-933b-5b1cab6b01f5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.056841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f0fe46a-561a-4d95-933b-5b1cab6b01f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.077446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2f0fe46a-561a-4d95-933b-5b1cab6b01f5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:16.078742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2f0fe46a-561a-4d95-933b-5b1cab6b01f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:16.087072 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 999/1995] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:16.092934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f838d1f-4b7b-4f87-8a4e-dee0b5e5f8d7 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:16.104232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f838d1f-4b7b-4f87-8a4e-dee0b5e5f8d7 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:16.109189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:16.109793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.110101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.110372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.111053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:16.111347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:16.112967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:16.125643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 997/1996] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:16.130102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0fe46a-561a-4d95-933b-5b1cab6b01f5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:16.140397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f0fe46a-561a-4d95-933b-5b1cab6b01f5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:16.145322 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:16.145929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.146241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.146512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.147137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:16.149051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:16.180271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1000/1997] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:16 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:16.184694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:16.195367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-616ed016-2b58-426b-99d2-cfa5792d7f7c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:16.253334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:16.254011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.254295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.254560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.255210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:16.257164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:16.280988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 998/1998] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:16 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:16.285642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:16.296663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1555311f-33d1-46f6-9b7d-3c988600d71d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:16.301669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:16.302382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.302726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.303065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.303821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:16.305437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:16.316078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1001/1999] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:16.320217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:16.333004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48df4744-1ab2-4ce4-ad6f-04d194d57747 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:16.337901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-377fa038-80a1-4214-9360-1dd917e50b51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:16.338752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377fa038-80a1-4214-9360-1dd917e50b51 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.339192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377fa038-80a1-4214-9360-1dd917e50b51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.339574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-377fa038-80a1-4214-9360-1dd917e50b51 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.340295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377fa038-80a1-4214-9360-1dd917e50b51 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:16.341978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-377fa038-80a1-4214-9360-1dd917e50b51 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:16.357673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 999/2000] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:16 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:16.358949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:16.369648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-492c027f-ef98-4719-82be-8385613ac711 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:16.374424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:16.375079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.375352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.375622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] PATH_INFO: `/v3/projects/d86f1088ae3245aea97c5c45fee351d1/users/f0103c58e7524d119ba9d664893e5714/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.390778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] RBAC: Authorizing `identity:create_grant(project_id=d86f1088ae3245aea97c5c45fee351d1, user_id=f0103c58e7524d119ba9d664893e5714, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:16.393721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:16.409033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1002/2001] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:16 2022] PUT /identity/v3/projects/d86f1088ae3245aea97c5c45fee351d1/users/f0103c58e7524d119ba9d664893e5714/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:16.421438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10702f6c-5ab6-45bf-bafb-34e24c7a97a7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.421736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10702f6c-5ab6-45bf-bafb-34e24c7a97a7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.422064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10702f6c-5ab6-45bf-bafb-34e24c7a97a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.453866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-10702f6c-5ab6-45bf-bafb-34e24c7a97a7 None None] MFA Rules not processed for user `f0103c58e7524d119ba9d664893e5714`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:16.455227 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-10702f6c-5ab6-45bf-bafb-34e24c7a97a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:16.509540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:16.520833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1000/2002] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:16 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:16.532938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:16.534918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:16.620531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5961f1e-4ab8-43a9-ae0f-4e803687f5bd admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:16.628463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1381396a-7d1d-41c8-aeb6-2a32f7b01564 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:16.629670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1381396a-7d1d-41c8-aeb6-2a32f7b01564 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:16.630113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1381396a-7d1d-41c8-aeb6-2a32f7b01564 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:16.630483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1381396a-7d1d-41c8-aeb6-2a32f7b01564 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:16.643870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1381396a-7d1d-41c8-aeb6-2a32f7b01564 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:16.645598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1381396a-7d1d-41c8-aeb6-2a32f7b01564 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:16.670827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1003/2003] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:16 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:17.654904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421eb189-8679-4628-ba1a-3d429c20fac3 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:17.655448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421eb189-8679-4628-ba1a-3d429c20fac3 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:17.655741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421eb189-8679-4628-ba1a-3d429c20fac3 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:17.657526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1001/2004] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:14:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:14:17.672333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df7cac0-9c02-4d19-9db0-bfd792d935cf None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:17.672905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df7cac0-9c02-4d19-9db0-bfd792d935cf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:17.672905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df7cac0-9c02-4d19-9db0-bfd792d935cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:17.692568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7df7cac0-9c02-4d19-9db0-bfd792d935cf None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:17.693780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7df7cac0-9c02-4d19-9db0-bfd792d935cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:17.802970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1004/2005] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:14:17 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:17.817309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421eb189-8679-4628-ba1a-3d429c20fac3 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:17.819805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-421eb189-8679-4628-ba1a-3d429c20fac3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:17.837389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421eb189-8679-4628-ba1a-3d429c20fac3 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:17.842968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fcbc567-85c4-475a-b8bb-f8a7b9f43802 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:17.843658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcbc567-85c4-475a-b8bb-f8a7b9f43802 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:17.843997 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcbc567-85c4-475a-b8bb-f8a7b9f43802 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:17.844329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fcbc567-85c4-475a-b8bb-f8a7b9f43802 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:17.854728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcbc567-85c4-475a-b8bb-f8a7b9f43802 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:17.856132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fcbc567-85c4-475a-b8bb-f8a7b9f43802 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:17.867827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1002/2006] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:17 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:18.368156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ad363-d1fd-4084-bb00-75fcd13c284d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.368551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ad363-d1fd-4084-bb00-75fcd13c284d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.368834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7ad363-d1fd-4084-bb00-75fcd13c284d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.370257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1005/2007] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:14:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:14:18.401888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95bf882-88f5-4788-b72a-849ad77bb81d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.402333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95bf882-88f5-4788-b72a-849ad77bb81d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.402604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95bf882-88f5-4788-b72a-849ad77bb81d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.425870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c95bf882-88f5-4788-b72a-849ad77bb81d None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:18.427319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c95bf882-88f5-4788-b72a-849ad77bb81d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.442282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ad363-d1fd-4084-bb00-75fcd13c284d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:18.442441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1003/2008] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:14:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:18.444416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8f7ad363-d1fd-4084-bb00-75fcd13c284d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.464392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e1b22e-7532-4b3c-a2d6-9c4ce5e0c70a None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.464392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e1b22e-7532-4b3c-a2d6-9c4ce5e0c70a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.464392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e1b22e-7532-4b3c-a2d6-9c4ce5e0c70a None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.464392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1004/2009] 10.222.0.61 () {54 vars in 961 bytes} [Thu Sep 29 02:14:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:14:18.467702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a706a74d-6790-47f2-91ee-02ec2c8bd7be None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.468219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a706a74d-6790-47f2-91ee-02ec2c8bd7be None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.468555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a706a74d-6790-47f2-91ee-02ec2c8bd7be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.474261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=3,keystone:SELECT=95,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:18.493259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a706a74d-6790-47f2-91ee-02ec2c8bd7be None None] MFA Rules not processed for user `8083af87f4f742be8b769d8b0bb7ca6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:18.494547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a706a74d-6790-47f2-91ee-02ec2c8bd7be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:18.509970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1005/2010] 10.222.0.61 () {58 vars in 1058 bytes} [Thu Sep 29 02:14:18 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:18.520735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a706a74d-6790-47f2-91ee-02ec2c8bd7be None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:18.527836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7ad363-d1fd-4084-bb00-75fcd13c284d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:18.535288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a706a74d-6790-47f2-91ee-02ec2c8bd7be None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:18.535775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:18.536557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.537001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.538652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] PATH_INFO: `/v3/users/f49c6293f9594461903218002a9386b9` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.538782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f49c6293f9594461903218002a9386b9)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:18.540825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:18.547403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:18.552795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.553060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.553327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.562635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:18.564274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:18.575778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1006/2011] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:18 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 56 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:18.614661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f49c6293f9594461903218002a9386b9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.622075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f49c6293f9594461903218002a9386b9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.622858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f49c6293f9594461903218002a9386b9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.623501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f49c6293f9594461903218002a9386b9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.632086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f49c6293f9594461903218002a9386b9'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:18.653686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1006/2012] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:18 2022] DELETE /identity/v3/users/f49c6293f9594461903218002a9386b9 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:18.863542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:18.875076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992d0d63-99ca-4620-9ef8-aea7c88ded31 service placement] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:18.961802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:18.963355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:18.963854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:18.964224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] PATH_INFO: `/v3/projects/d39411cc4b0a4d00997bebdef0c210b7` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:18.965515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d39411cc4b0a4d00997bebdef0c210b7)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:18.967703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.019641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd39411cc4b0a4d00997bebdef0c210b7', 'request_id': 'req-36cc40c2-316b-4cfd-8c9a-41341092479b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.029260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd39411cc4b0a4d00997bebdef0c210b7', 'request_id': 'req-36cc40c2-316b-4cfd-8c9a-41341092479b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.036849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd39411cc4b0a4d00997bebdef0c210b7', 'request_id': 'req-36cc40c2-316b-4cfd-8c9a-41341092479b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.048473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] The token cache is being invalidate because project d39411cc4b0a4d00997bebdef0c210b7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:19.048473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.049046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.050632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1007/2013] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:18 2022] DELETE /identity/v3/projects/d39411cc4b0a4d00997bebdef0c210b7 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:19.054831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.056398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.069750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521bf8d9-5ff1-4ba2-b39a-65fcacddc336 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.133760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.134436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.134714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.135632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] PATH_INFO: `/v3/users/72efe7d3b30e48fa94793ec712bb9370` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.136778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] RBAC: Authorizing `identity:delete_user(user_id=72efe7d3b30e48fa94793ec712bb9370)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.138744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.189958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:19.199881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '72efe7d3b30e48fa94793ec712bb9370'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.208233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '72efe7d3b30e48fa94793ec712bb9370'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.209430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '72efe7d3b30e48fa94793ec712bb9370'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.210079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '72efe7d3b30e48fa94793ec712bb9370'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.220850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '72efe7d3b30e48fa94793ec712bb9370'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.245399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1007/2014] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:19 2022] DELETE /identity/v3/users/72efe7d3b30e48fa94793ec712bb9370 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:19.325224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.326876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.339698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36cc40c2-316b-4cfd-8c9a-41341092479b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.382301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:19.399478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.400121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.400505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.400808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] PATH_INFO: `/v3/users/f0103c58e7524d119ba9d664893e5714` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.401913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0103c58e7524d119ba9d664893e5714)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.403904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.427480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.430896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.446083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fcb8c6d-5e4c-496d-abe6-f05bf37e231c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.451316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.451956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.452235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.452498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] PATH_INFO: `/v3/projects/7f2a1bc7984e42d39fa9fd78e3763711` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.453621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] RBAC: Authorizing `identity:delete_project(project_id=7f2a1bc7984e42d39fa9fd78e3763711)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.455942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.462037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0103c58e7524d119ba9d664893e5714'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.470527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0103c58e7524d119ba9d664893e5714'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.471347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0103c58e7524d119ba9d664893e5714'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.472028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0103c58e7524d119ba9d664893e5714'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.480818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0c585fc5-ec32-4ec1-9298-e71f0b869a38 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0103c58e7524d119ba9d664893e5714'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.499995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1008/2015] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:19 2022] DELETE /identity/v3/users/f0103c58e7524d119ba9d664893e5714 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:19.504686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.504974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.505249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.505626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7f2a1bc7984e42d39fa9fd78e3763711', 'request_id': 'req-cb6c5b3f-7901-436d-aa85-48f41dab9fce'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.514307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7f2a1bc7984e42d39fa9fd78e3763711', 'request_id': 'req-cb6c5b3f-7901-436d-aa85-48f41dab9fce'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.521075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7f2a1bc7984e42d39fa9fd78e3763711', 'request_id': 'req-cb6c5b3f-7901-436d-aa85-48f41dab9fce'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.525726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:19.526942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.533150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] The token cache is being invalidate because project 7f2a1bc7984e42d39fa9fd78e3763711 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:19.533376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.534489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-cb6c5b3f-7901-436d-aa85-48f41dab9fce admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:19.536209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1008/2016] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:19 2022] DELETE /identity/v3/projects/7f2a1bc7984e42d39fa9fd78e3763711 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:19.541210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce527a68-fbd6-41ea-8aff-7b4fd98dfe5e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.541559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce527a68-fbd6-41ea-8aff-7b4fd98dfe5e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.541831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce527a68-fbd6-41ea-8aff-7b4fd98dfe5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.562808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ce527a68-fbd6-41ea-8aff-7b4fd98dfe5e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:19.563955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ce527a68-fbd6-41ea-8aff-7b4fd98dfe5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.588521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1009/2017] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:19.598797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.600884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.628917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1009/2018] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:19.638853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eef576-dcd7-41af-9787-2d6df094a72c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.639190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eef576-dcd7-41af-9787-2d6df094a72c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.639505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25eef576-dcd7-41af-9787-2d6df094a72c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.659610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-25eef576-dcd7-41af-9787-2d6df094a72c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:19.661035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-25eef576-dcd7-41af-9787-2d6df094a72c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.665808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a33ca5-8316-4484-a407-92b31795f601 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.669910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.670441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1010/2019] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:19.670561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.670810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.671139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.674576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eef576-dcd7-41af-9787-2d6df094a72c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.681117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.682710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.683688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25eef576-dcd7-41af-9787-2d6df094a72c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.688460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.689044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.689447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.689647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.690315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:19.690617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.692217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.692613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1010/2020] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:19.709410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1011/2021] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:14:19.710244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.720943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a2ed3cd-a666-4374-a799-7caac74464be service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.726582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.727207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.727480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.727771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.728379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.730570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.735612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:19.765513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1011/2022] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/projects => generated 385 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:19.767061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.778124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62efa0f7-6231-48f7-9417-d48b39d26db6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.791547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:19.849725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.850483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.850821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.851146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.851849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.853831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.876409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1012/2023] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:19 2022] POST /identity/v3/users => generated 293 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:19.880890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.892900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954b408d-fc97-4a3e-91e4-18e0eb36d5a6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.897685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.898302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.898578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.898846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.899489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.901101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.911487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1012/2024] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:19.915695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.926422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc15dec6-e288-4963-9c0d-891409bae9c3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.931211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.931818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.932095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.932479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.933129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.934783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.946608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1013/2025] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:19 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:19.951050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.962089 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc8f3f05-b97f-438c-b395-9f974c30b0f1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.967420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a22c35af-3bb8-4423-8888-6e18cb064538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.968424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22c35af-3bb8-4423-8888-6e18cb064538 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.968698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22c35af-3bb8-4423-8888-6e18cb064538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.968964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a22c35af-3bb8-4423-8888-6e18cb064538 admin admin] PATH_INFO: `/v3/projects/cff2f494a8064952a7ff8184ef8fea7f/users/ea4a2273af6a4ca48f09ca427849d66a/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.969073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:19.971463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:19.984805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985c5dbd-7fe4-42b5-b1a4-5dc929d1d982 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:19.987491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22c35af-3bb8-4423-8888-6e18cb064538 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cff2f494a8064952a7ff8184ef8fea7f, user_id=ea4a2273af6a4ca48f09ca427849d66a, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:19.991171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a22c35af-3bb8-4423-8888-6e18cb064538 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:19.995462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:19.996310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:19.996919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:19.997218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] PATH_INFO: `/v3/projects/d86f1088ae3245aea97c5c45fee351d1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:19.998558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d86f1088ae3245aea97c5c45fee351d1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.000765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.012764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1013/2026] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:19 2022] PUT /identity/v3/projects/cff2f494a8064952a7ff8184ef8fea7f/users/ea4a2273af6a4ca48f09ca427849d66a/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:20.023404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.024058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.024648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.057674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd86f1088ae3245aea97c5c45fee351d1', 'request_id': 'req-d8917e57-49d0-40c4-8e96-e4174ec55c00'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:20.064765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] MFA Rules not processed for user `ea4a2273af6a4ca48f09ca427849d66a`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:20.066654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.070834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd86f1088ae3245aea97c5c45fee351d1', 'request_id': 'req-d8917e57-49d0-40c4-8e96-e4174ec55c00'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:20.080956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd86f1088ae3245aea97c5c45fee351d1', 'request_id': 'req-d8917e57-49d0-40c4-8e96-e4174ec55c00'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:20.099464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] The token cache is being invalidate because project d86f1088ae3245aea97c5c45fee351d1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:20.099934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:20.100646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8917e57-49d0-40c4-8e96-e4174ec55c00 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:20.103104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1014/2027] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:19 2022] DELETE /identity/v3/projects/d86f1088ae3245aea97c5c45fee351d1 => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:20.107797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00ed81-6c46-4755-8c5a-1bc945157360 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.108074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00ed81-6c46-4755-8c5a-1bc945157360 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.108346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c00ed81-6c46-4755-8c5a-1bc945157360 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.132245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3c00ed81-6c46-4755-8c5a-1bc945157360 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:20.133640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3c00ed81-6c46-4755-8c5a-1bc945157360 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.146347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1014/2028] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:20.150384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.151925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.197787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1015/2029] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:20.205663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffa5709-c655-4e0d-9f72-8bf5c727bb2c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.206057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffa5709-c655-4e0d-9f72-8bf5c727bb2c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.206339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffa5709-c655-4e0d-9f72-8bf5c727bb2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.225956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4ffa5709-c655-4e0d-9f72-8bf5c727bb2c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:20.227259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4ffa5709-c655-4e0d-9f72-8bf5c727bb2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.235290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1016/2030] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:20.235831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ef51fe5-e5fa-4249-91fa-13f4b78270b6 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.239609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffa5709-c655-4e0d-9f72-8bf5c727bb2c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.241725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.242375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.242648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.242911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.243540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.245450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.248562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffa5709-c655-4e0d-9f72-8bf5c727bb2c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.254667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.255297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.255538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.255806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.256444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:20.256739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.258381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.269043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:20.274356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1017/2031] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:20.278048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1015/2032] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/projects => generated 383 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:20.279189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.280918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.290233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13bd9ad-c1db-4b5e-a93b-dafbcab3fb6a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.308297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:20.353800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d5d043d-4e7b-4808-a797-1e6124c875d0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.355315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.355936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.356209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.356513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.357105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.359152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.359526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.360151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.360413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.360674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.361628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.363733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.390375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1016/2033] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/users => generated 298 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:20.394504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.402037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1018/2034] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/projects => generated 397 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:20.403866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.407839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2febf2d-3e29-4355-a8bc-9da940d799b9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.473327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.474066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.474399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.474724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.475438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.475555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895b4910-7a32-45ef-b10e-1a71e06c40ba admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.477165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.481936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.482991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.483269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.483539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.484171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.486499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.491442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1017/2035] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:20.494768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.505194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030aa7dd-c156-4874-a410-be0447e743ed admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.510099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.510769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.511044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.511317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] PATH_INFO: `/v3/projects/0e3cda5f16b5491eaee9a2467e5d7453/users/b220b25271564d4dbaaffd0be12d5a04/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.512845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1019/2036] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/users => generated 299 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:20.517226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.529894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e3cda5f16b5491eaee9a2467e5d7453, user_id=b220b25271564d4dbaaffd0be12d5a04, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.532856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-224134b7-2ffb-47e0-b86b-eaf6f685971a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.533092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.538425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.539285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.539769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.540296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.541149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.543171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.555486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1018/2037] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:20 2022] PUT /identity/v3/projects/0e3cda5f16b5491eaee9a2467e5d7453/users/b220b25271564d4dbaaffd0be12d5a04/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:20.556882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1020/2038] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:20.558523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.560867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.571735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c886455d-4a17-4b15-9ba1-65c49de128eb admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.630918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d7b5661-9bc2-474a-91d0-f334e6d8bcc2 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.636087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.636776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.637097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.637229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.637492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.637710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.638086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.638363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.638467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.639007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.640140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.640654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.646873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:20.653619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1019/2039] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:20.656172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1021/2040] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:20.656750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.657210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.667329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3408cf7-0e98-4f6f-a1c8-30464c23edc9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.668717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc89c27-1a30-4cdb-9dd6-9b219555fcba admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.672037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0d9cdf-cb11-481d-abcb-6a0da3ecbf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.672825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0d9cdf-cb11-481d-abcb-6a0da3ecbf73 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.673107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0d9cdf-cb11-481d-abcb-6a0da3ecbf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.673381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0d9cdf-cb11-481d-abcb-6a0da3ecbf73 admin admin] PATH_INFO: `/v3/projects/0e3cda5f16b5491eaee9a2467e5d7453/users/b220b25271564d4dbaaffd0be12d5a04/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.674203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69201521-f9b4-4bbb-aaa7-a4626be0f214 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.675100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69201521-f9b4-4bbb-aaa7-a4626be0f214 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.675430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0d9cdf-cb11-481d-abcb-6a0da3ecbf73 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0e3cda5f16b5491eaee9a2467e5d7453, user_id=b220b25271564d4dbaaffd0be12d5a04, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.675798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69201521-f9b4-4bbb-aaa7-a4626be0f214 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.676293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69201521-f9b4-4bbb-aaa7-a4626be0f214 admin admin] PATH_INFO: `/v3/projects/7214321065d8499fa761add95014d08a/users/5163e25dd8af43cbac504edc679d12c0/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.678180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0d9cdf-cb11-481d-abcb-6a0da3ecbf73 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.702838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1020/2041] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:20 2022] PUT /identity/v3/projects/0e3cda5f16b5491eaee9a2467e5d7453/users/b220b25271564d4dbaaffd0be12d5a04/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:20.705788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69201521-f9b4-4bbb-aaa7-a4626be0f214 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7214321065d8499fa761add95014d08a, user_id=5163e25dd8af43cbac504edc679d12c0, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.708924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69201521-f9b4-4bbb-aaa7-a4626be0f214 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.713801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.714232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.714539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.735020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1022/2042] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:20 2022] PUT /identity/v3/projects/7214321065d8499fa761add95014d08a/users/5163e25dd8af43cbac504edc679d12c0/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:20.738582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.738936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.739184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:20.739462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.750392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] MFA Rules not processed for user `b220b25271564d4dbaaffd0be12d5a04`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:20.752477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.773023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] MFA Rules not processed for user `5163e25dd8af43cbac504edc679d12c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:20.774761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.825465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1021/2043] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:20.836667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.838112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.850340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1023/2044] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:20 2022] POST /identity/v3/auth/tokens => generated 3274 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:20.862052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:20.863489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:20.903911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2669cd-fcd1-48a9-93fa-a14151d9824e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.907588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.908342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.908692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.909036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.923762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f704fb-7fe3-4c39-96b5-539597648f6b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:20.928158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:20.929186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:20.929746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:20.929746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:20.939463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.940997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.950672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1024/2045] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/auth/tokens => generated 3274 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:20.971568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:20.976908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:20.983500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1022/2046] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:20 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:24.213608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:24.216262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:24.318402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea606bc-d19e-4422-96ce-c71b4e098c6e service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:24.323369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:24.323903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:24.324147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:24.324426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] PATH_INFO: `/v3/users/3b624208bb94427791219ffcca0efc41` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:24.325663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b624208bb94427791219ffcca0efc41)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:24.327659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:24.340471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:24.394260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b624208bb94427791219ffcca0efc41'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.402684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b624208bb94427791219ffcca0efc41'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.403398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b624208bb94427791219ffcca0efc41'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.404062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b624208bb94427791219ffcca0efc41'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.426466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7e01c4d7-7db9-4583-821d-82e105b20d3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b624208bb94427791219ffcca0efc41'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.447556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1025/2047] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:24 2022] DELETE /identity/v3/users/3b624208bb94427791219ffcca0efc41 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:24.635064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:24.646534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03e9fb9-5dc3-438b-8588-924b1828a937 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:24.692103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:24.704458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:24.705326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:24.705863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:24.705863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] PATH_INFO: `/v3/projects/45259b9967b9410ca0fb09df7cea9f15` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:24.707007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] RBAC: Authorizing `identity:delete_project(project_id=45259b9967b9410ca0fb09df7cea9f15)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:24.708912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:24.757679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '45259b9967b9410ca0fb09df7cea9f15', 'request_id': 'req-0a5a0712-c934-4831-93fd-41115b1bc93b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.768473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '45259b9967b9410ca0fb09df7cea9f15', 'request_id': 'req-0a5a0712-c934-4831-93fd-41115b1bc93b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.775279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '45259b9967b9410ca0fb09df7cea9f15', 'request_id': 'req-0a5a0712-c934-4831-93fd-41115b1bc93b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.785557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] The token cache is being invalidate because project 45259b9967b9410ca0fb09df7cea9f15 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:24.785837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.786479 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0a5a0712-c934-4831-93fd-41115b1bc93b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:24.787983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1023/2048] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:24 2022] DELETE /identity/v3/projects/45259b9967b9410ca0fb09df7cea9f15 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:24.799617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c16f53f-ae0c-414d-914b-456cba5f1f81 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:24.799904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c16f53f-ae0c-414d-914b-456cba5f1f81 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:24.800181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c16f53f-ae0c-414d-914b-456cba5f1f81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:24.822942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2c16f53f-ae0c-414d-914b-456cba5f1f81 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:24.824171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2c16f53f-ae0c-414d-914b-456cba5f1f81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:24.888329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1026/2049] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:24.896603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58412b9-4e7c-4380-8fc4-4757bbad696e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:24.896885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58412b9-4e7c-4380-8fc4-4757bbad696e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:24.897161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b58412b9-4e7c-4380-8fc4-4757bbad696e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:24.916813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b58412b9-4e7c-4380-8fc4-4757bbad696e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:24.918033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b58412b9-4e7c-4380-8fc4-4757bbad696e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:24.926510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1024/2050] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:24.930918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c16f53f-ae0c-414d-914b-456cba5f1f81 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:24.940968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c16f53f-ae0c-414d-914b-456cba5f1f81 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:24.945885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:24.946526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:24.946793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:24.947059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:24.947699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:24.947992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:24.949644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:24.962540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1027/2051] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:24.967093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58412b9-4e7c-4380-8fc4-4757bbad696e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:24.977075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b58412b9-4e7c-4380-8fc4-4757bbad696e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:24.981816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:24.982466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:24.982734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:24.982999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:24.983820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:24.985747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.025764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1025/2052] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:24 2022] POST /identity/v3/projects => generated 363 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:25.030375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.043724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c19bb74-879d-4a32-a523-09c0dfb827c3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.114886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.115368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.115665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.115934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.116569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.118618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.141785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1028/2053] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:25 2022] POST /identity/v3/users => generated 282 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:25.146773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.157007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71e72c79-70c1-4a1d-94d6-daedee31d9ca admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.161786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.162598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.162873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.163139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.163775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.165496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.175594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1026/2054] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:25.179714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.186300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:25.190668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3498b9-0702-49c4-865e-32191ff58ffa admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.197430 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db29661-bfa0-4469-a3c8-580a640b83c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.198138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db29661-bfa0-4469-a3c8-580a640b83c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.198463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db29661-bfa0-4469-a3c8-580a640b83c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.198741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db29661-bfa0-4469-a3c8-580a640b83c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.199400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db29661-bfa0-4469-a3c8-580a640b83c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.201070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db29661-bfa0-4469-a3c8-580a640b83c7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.212702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1029/2055] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:25.216880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.227746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fe6cac5-007f-426f-bbbf-5428a210888b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.233147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.233803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.234107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.234381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] PATH_INFO: `/v3/projects/5ae06d4700e445d79a70e9fd8004b5a9/users/978331a4100641168ef60b1f493a8388/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.250055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5ae06d4700e445d79a70e9fd8004b5a9, user_id=978331a4100641168ef60b1f493a8388, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.252919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.269243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1027/2056] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:25 2022] PUT /identity/v3/projects/5ae06d4700e445d79a70e9fd8004b5a9/users/978331a4100641168ef60b1f493a8388/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:25.279542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53edb5a-d030-4b6a-9b3d-204c9cf5b942 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.280014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53edb5a-d030-4b6a-9b3d-204c9cf5b942 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.280106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53edb5a-d030-4b6a-9b3d-204c9cf5b942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.309584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a53edb5a-d030-4b6a-9b3d-204c9cf5b942 None None] MFA Rules not processed for user `978331a4100641168ef60b1f493a8388`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:25.310852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a53edb5a-d030-4b6a-9b3d-204c9cf5b942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:25.381529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1030/2057] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:25 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:25.386373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.452606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9d48741-ca44-4e50-affa-30fc7e5a7332 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.457763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.458411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.458690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.458962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.459594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.461508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.486391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:25.497882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1028/2058] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:25 2022] POST /identity/v3/projects => generated 363 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:25.499084 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53edb5a-d030-4b6a-9b3d-204c9cf5b942 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.510466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a53edb5a-d030-4b6a-9b3d-204c9cf5b942 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.570315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.570927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.571203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.571472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.572095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.574042 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.600227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1031/2059] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:25 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:25.604843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.618335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca18b204-41e3-483b-bdae-b78fae1c9dee admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.626310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.627376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.627656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.627922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.628564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.630222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.641335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1029/2060] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:25.645718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.655954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8cc5a77-f4dc-492c-85bf-27fa72b5c4db admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.660840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.661488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.661762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.662057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] PATH_INFO: `/v3/projects/6e189889147e49a2b6f6565c22771e9e/users/79cdb84163cd4a99a99959615ff039cd/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.677058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e189889147e49a2b6f6565c22771e9e, user_id=79cdb84163cd4a99a99959615ff039cd, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.679769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.697381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1032/2061] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:25 2022] PUT /identity/v3/projects/6e189889147e49a2b6f6565c22771e9e/users/79cdb84163cd4a99a99959615ff039cd/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:25.701262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.713328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e403d090-039f-43bb-8ba3-c310d364af7a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.780574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ab995b-baf7-416f-94fc-b26b1478ef95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.781390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ab995b-baf7-416f-94fc-b26b1478ef95 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.781764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ab995b-baf7-416f-94fc-b26b1478ef95 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.782172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ab995b-baf7-416f-94fc-b26b1478ef95 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.783129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ab995b-baf7-416f-94fc-b26b1478ef95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.784920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ab995b-baf7-416f-94fc-b26b1478ef95 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.794509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1030/2062] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:25 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:25.799424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.810455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2bafcca-e350-4955-9e7d-8d268b07fd00 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:25.816733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:25.817447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.817719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.817981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] PATH_INFO: `/v3/projects/6e189889147e49a2b6f6565c22771e9e/users/79cdb84163cd4a99a99959615ff039cd/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.820139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e189889147e49a2b6f6565c22771e9e, user_id=79cdb84163cd4a99a99959615ff039cd, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:25.822856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:25.834510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1033/2063] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:25 2022] PUT /identity/v3/projects/6e189889147e49a2b6f6565c22771e9e/users/79cdb84163cd4a99a99959615ff039cd/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:25.847109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6f7a4-75e7-4bbb-b319-1ed1a8fb0fe4 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:25.847391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6f7a4-75e7-4bbb-b319-1ed1a8fb0fe4 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:25.847662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c6f7a4-75e7-4bbb-b319-1ed1a8fb0fe4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:25.878350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-44c6f7a4-75e7-4bbb-b319-1ed1a8fb0fe4 None None] MFA Rules not processed for user `79cdb84163cd4a99a99959615ff039cd`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:25.879560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-44c6f7a4-75e7-4bbb-b319-1ed1a8fb0fe4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:25.960049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1031/2064] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:25 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:25.974955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:25.977033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:26.020359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:26.041283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e17a7e-b192-4514-b8b2-37d103349cf1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:26.045420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e97f5a9-8479-486d-a780-366fa3d1f624 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:26.046033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e97f5a9-8479-486d-a780-366fa3d1f624 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:26.046330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e97f5a9-8479-486d-a780-366fa3d1f624 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:26.046601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e97f5a9-8479-486d-a780-366fa3d1f624 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:26.056439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e97f5a9-8479-486d-a780-366fa3d1f624 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:26.057841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e97f5a9-8479-486d-a780-366fa3d1f624 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:26.068021 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1034/2065] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:25 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:26.518430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c6f7a4-75e7-4bbb-b319-1ed1a8fb0fe4 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:26.581130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c6f7a4-75e7-4bbb-b319-1ed1a8fb0fe4 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:26.585645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:26.586310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:26.586582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:26.586852 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] PATH_INFO: `/v3/users/978331a4100641168ef60b1f493a8388` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:26.587959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] RBAC: Authorizing `identity:delete_user(user_id=978331a4100641168ef60b1f493a8388)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:26.589883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:26.602489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:26.647743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '978331a4100641168ef60b1f493a8388'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:26.658275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '978331a4100641168ef60b1f493a8388'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:26.659187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '978331a4100641168ef60b1f493a8388'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:26.659873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '978331a4100641168ef60b1f493a8388'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:26.669468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '978331a4100641168ef60b1f493a8388'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:26.687438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1032/2066] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:26 2022] DELETE /identity/v3/users/978331a4100641168ef60b1f493a8388 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:26.692700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:26.693172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:26.693265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:26.712473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:26.713669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:26.777699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1035/2067] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:26.790932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:26.795672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:26.859744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972b63a4-f548-42e3-83c6-ea4b7da97c08 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:26.864064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:26.865072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:26.865344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:26.865609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:26.875026 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:26.876495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:26.885933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1033/2068] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:27.171559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:27.173179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:27.185427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b02d7221-f861-4dee-9bed-116806a59b2f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:27.190489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:27.191145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:27.191420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:27.191737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] PATH_INFO: `/v3/projects/5ae06d4700e445d79a70e9fd8004b5a9` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:27.192974 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] RBAC: Authorizing `identity:delete_project(project_id=5ae06d4700e445d79a70e9fd8004b5a9)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:27.195038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:27.246267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5ae06d4700e445d79a70e9fd8004b5a9', 'request_id': 'req-156258aa-da33-43c7-8a0e-c841a457431e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.255578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5ae06d4700e445d79a70e9fd8004b5a9', 'request_id': 'req-156258aa-da33-43c7-8a0e-c841a457431e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.262681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5ae06d4700e445d79a70e9fd8004b5a9', 'request_id': 'req-156258aa-da33-43c7-8a0e-c841a457431e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.274745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] The token cache is being invalidate because project 5ae06d4700e445d79a70e9fd8004b5a9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:27.275031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.275714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.277294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1036/2069] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:27 2022] DELETE /identity/v3/projects/5ae06d4700e445d79a70e9fd8004b5a9 => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:27.281509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:27.291984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6954bdeb-8523-46fe-abe0-ccf91077494a service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:27.354136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:27.354776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:27.355053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:27.355320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] PATH_INFO: `/v3/users/79cdb84163cd4a99a99959615ff039cd` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:27.356629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] RBAC: Authorizing `identity:delete_user(user_id=79cdb84163cd4a99a99959615ff039cd)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:27.358786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:27.400653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=85,keystone:DELETE=14,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:27.413099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '79cdb84163cd4a99a99959615ff039cd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.420053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '79cdb84163cd4a99a99959615ff039cd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.420740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '79cdb84163cd4a99a99959615ff039cd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.421351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '79cdb84163cd4a99a99959615ff039cd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.430773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b3da077-022f-4df1-8469-220ee03f931a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '79cdb84163cd4a99a99959615ff039cd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.450752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1034/2070] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:27 2022] DELETE /identity/v3/users/79cdb84163cd4a99a99959615ff039cd => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:27.728680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:27.730181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:27.745578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-156258aa-da33-43c7-8a0e-c841a457431e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:27.806519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:27.807158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:27.807434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:27.807704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] PATH_INFO: `/v3/projects/6e189889147e49a2b6f6565c22771e9e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:27.808893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e189889147e49a2b6f6565c22771e9e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:27.810866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:27.813300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:27.861627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e189889147e49a2b6f6565c22771e9e', 'request_id': 'req-5e4bfa99-64fc-4295-8d2f-056493e70d8d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.871494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e189889147e49a2b6f6565c22771e9e', 'request_id': 'req-5e4bfa99-64fc-4295-8d2f-056493e70d8d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.879218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e189889147e49a2b6f6565c22771e9e', 'request_id': 'req-5e4bfa99-64fc-4295-8d2f-056493e70d8d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.890626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] The token cache is being invalidate because project 6e189889147e49a2b6f6565c22771e9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:27.890943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.891675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5e4bfa99-64fc-4295-8d2f-056493e70d8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:27.893275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1037/2071] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:27 2022] DELETE /identity/v3/projects/6e189889147e49a2b6f6565c22771e9e => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:27.898500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e310ee00-0990-4eb8-8fcc-ae1ed77efaa8 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:27.899011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e310ee00-0990-4eb8-8fcc-ae1ed77efaa8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:27.899011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e310ee00-0990-4eb8-8fcc-ae1ed77efaa8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:27.920614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-e310ee00-0990-4eb8-8fcc-ae1ed77efaa8 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:27.921749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e310ee00-0990-4eb8-8fcc-ae1ed77efaa8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:27.986984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1035/2072] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:27.999424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff7a650-87b6-4509-8cc8-bca306b40ff0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:27.999789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff7a650-87b6-4509-8cc8-bca306b40ff0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.000147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fff7a650-87b6-4509-8cc8-bca306b40ff0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.021503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fff7a650-87b6-4509-8cc8-bca306b40ff0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:28.022750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fff7a650-87b6-4509-8cc8-bca306b40ff0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:28.032149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1038/2073] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:28.036831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e310ee00-0990-4eb8-8fcc-ae1ed77efaa8 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.047616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e310ee00-0990-4eb8-8fcc-ae1ed77efaa8 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.053591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.054235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.054511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.054772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.055438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:28.055852 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.057459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.069727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1036/2074] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:28.074438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff7a650-87b6-4509-8cc8-bca306b40ff0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.084347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fff7a650-87b6-4509-8cc8-bca306b40ff0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.090274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.090866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.091140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.091408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.092041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.093947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.130331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1039/2075] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:28 2022] POST /identity/v3/projects => generated 377 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:28.131680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.141947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cb17a58-48ff-488f-8778-e5667215d962 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.203555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.204083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.204357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.204627 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.205251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.207209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.231804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1037/2076] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:28 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:28.236529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.246855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-876963fb-b52f-41c6-b4a3-62ab59a80433 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.251393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.251987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.252431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.252696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.253337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.254980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.265494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1040/2077] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:28.270141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.280309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479ffb6e-5dcf-4179-a9fe-1bc105c1f35e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.285897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39016568-99e9-44cc-b967-40b45bf63202 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.286526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39016568-99e9-44cc-b967-40b45bf63202 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.286822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39016568-99e9-44cc-b967-40b45bf63202 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.287087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39016568-99e9-44cc-b967-40b45bf63202 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.287721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39016568-99e9-44cc-b967-40b45bf63202 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.289317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39016568-99e9-44cc-b967-40b45bf63202 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.300431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1038/2078] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:28.304733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.315248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b69166e-cd5a-4b93-a05d-bb78a0d811ed admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.320624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.321265 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.321536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.321801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] PATH_INFO: `/v3/projects/6a7d698878b044c6b458b92d61b7c7a3/users/265039311b86450ba4c3addc6f17c505/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.337537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6a7d698878b044c6b458b92d61b7c7a3, user_id=265039311b86450ba4c3addc6f17c505, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.340230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.355378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1041/2079] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:28 2022] PUT /identity/v3/projects/6a7d698878b044c6b458b92d61b7c7a3/users/265039311b86450ba4c3addc6f17c505/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:28.366982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5df77d-adff-4f43-80aa-2da229f371ad None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.367260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5df77d-adff-4f43-80aa-2da229f371ad None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.367526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c5df77d-adff-4f43-80aa-2da229f371ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.395245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1c5df77d-adff-4f43-80aa-2da229f371ad None None] MFA Rules not processed for user `265039311b86450ba4c3addc6f17c505`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:28.396433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1c5df77d-adff-4f43-80aa-2da229f371ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:28.401734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:28.475334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1039/2080] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:28 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:28.480018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.546373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdabb454-6d30-401b-ae8f-787653082212 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.552201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.552969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.553254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.553526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.554802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.556429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.589401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1042/2081] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:28 2022] POST /identity/v3/projects => generated 375 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:28.594256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5df77d-adff-4f43-80aa-2da229f371ad None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.607152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c5df77d-adff-4f43-80aa-2da229f371ad None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.665718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.666357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.666893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.667166 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.667789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.669705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.693272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1040/2082] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:28 2022] POST /identity/v3/users => generated 294 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:28.697488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.708444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27213973-f8e7-4fa8-ac8c-1004244e900a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.713840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.714478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.714869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.715146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.715820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.717477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.728668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1043/2083] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:28.733001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.743727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4d66488-c8f8-4bf0-b092-c702aaf8d095 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.748801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.749454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.749724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.749991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] PATH_INFO: `/v3/projects/a381d1a27fa04724b00988146ffe0d41/users/60b0177f26104516bbd60942954a7b65/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.765770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] RBAC: Authorizing `identity:create_grant(project_id=a381d1a27fa04724b00988146ffe0d41, user_id=60b0177f26104516bbd60942954a7b65, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.768548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.784374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1041/2084] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:28 2022] PUT /identity/v3/projects/a381d1a27fa04724b00988146ffe0d41/users/60b0177f26104516bbd60942954a7b65/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:28.788248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.793478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:28.799712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71d8b8b4-051b-4b5b-b899-ec8e14d6436d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.864712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f471e10e-eca5-40aa-a71d-dda226436dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.865314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471e10e-eca5-40aa-a71d-dda226436dae admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.865582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471e10e-eca5-40aa-a71d-dda226436dae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.865853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f471e10e-eca5-40aa-a71d-dda226436dae admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.866523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f471e10e-eca5-40aa-a71d-dda226436dae admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.868130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f471e10e-eca5-40aa-a71d-dda226436dae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.879041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1044/2085] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:28 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:28.883999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:28.900921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e39c2463-8428-410d-8166-4b7873540bfb admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:28.910284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:28.910935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.911207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.911476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] PATH_INFO: `/v3/projects/a381d1a27fa04724b00988146ffe0d41/users/60b0177f26104516bbd60942954a7b65/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.913681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a381d1a27fa04724b00988146ffe0d41, user_id=60b0177f26104516bbd60942954a7b65, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:28.916401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:28.927180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1042/2086] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:28 2022] PUT /identity/v3/projects/a381d1a27fa04724b00988146ffe0d41/users/60b0177f26104516bbd60942954a7b65/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:28.937462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dc87f9-6aec-4db6-8494-7be1ae1c9ff1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:28.937743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dc87f9-6aec-4db6-8494-7be1ae1c9ff1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:28.938039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dc87f9-6aec-4db6-8494-7be1ae1c9ff1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:28.967744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-70dc87f9-6aec-4db6-8494-7be1ae1c9ff1 None None] MFA Rules not processed for user `60b0177f26104516bbd60942954a7b65`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:28.968820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-70dc87f9-6aec-4db6-8494-7be1ae1c9ff1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:29.041013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1045/2087] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:28 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:29.055067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:29.057050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:29.119373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78c63ab0-1d91-445c-89e0-fecc91100e96 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:29.122989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:29.123584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:29.123853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:29.124114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:29.136315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:29.187588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:29.189075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:29.198738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1043/2088] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:29 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:29.780562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dc87f9-6aec-4db6-8494-7be1ae1c9ff1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:29.791025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dc87f9-6aec-4db6-8494-7be1ae1c9ff1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:29.794494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9319a393-b242-41b2-bc89-9fca848e4031 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:29.795102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319a393-b242-41b2-bc89-9fca848e4031 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:29.795377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319a393-b242-41b2-bc89-9fca848e4031 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:29.795649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9319a393-b242-41b2-bc89-9fca848e4031 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:29.797442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9319a393-b242-41b2-bc89-9fca848e4031 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:29.862330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9319a393-b242-41b2-bc89-9fca848e4031 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:29.863777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9319a393-b242-41b2-bc89-9fca848e4031 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:29.874705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1046/2089] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:29 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:31.521187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:31.522726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:31.592011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f196eb-0cb3-464e-8a98-af289c8b6e1f service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:31.597405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:31.598100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:31.598398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:31.598630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] PATH_INFO: `/v3/users/5163e25dd8af43cbac504edc679d12c0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:31.599767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] RBAC: Authorizing `identity:delete_user(user_id=5163e25dd8af43cbac504edc679d12c0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:31.601728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:31.655562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5163e25dd8af43cbac504edc679d12c0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:31.663289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5163e25dd8af43cbac504edc679d12c0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:31.663977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5163e25dd8af43cbac504edc679d12c0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:31.664592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5163e25dd8af43cbac504edc679d12c0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:31.675360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5163e25dd8af43cbac504edc679d12c0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:31.695567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1044/2090] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:31 2022] DELETE /identity/v3/users/5163e25dd8af43cbac504edc679d12c0 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:31.700811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:31.701306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:31.701509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:31.724936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:31.726205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:31.738963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:31.794358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1047/2091] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:31.805544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:31.807192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:31.853534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:31.877597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be188614-4cde-4f85-a417-81d739ead48e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:31.881969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a85103a5-9df1-4aa5-9007-e1637d7bba41 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:31.882607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85103a5-9df1-4aa5-9007-e1637d7bba41 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:31.882882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85103a5-9df1-4aa5-9007-e1637d7bba41 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:31.883151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a85103a5-9df1-4aa5-9007-e1637d7bba41 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:31.891994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85103a5-9df1-4aa5-9007-e1637d7bba41 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:31.893432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a85103a5-9df1-4aa5-9007-e1637d7bba41 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:31.909034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1045/2092] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:32.269663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:32.271164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:32.282980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4488f340-57f0-4235-a472-21ec65f27ebd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:32.287518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:32.288140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.288484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.288754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] PATH_INFO: `/v3/projects/7214321065d8499fa761add95014d08a` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.289893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7214321065d8499fa761add95014d08a)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:32.291816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:32.340907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7214321065d8499fa761add95014d08a', 'request_id': 'req-d8958591-373c-41a3-919c-0fafe7a80764'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:32.350720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7214321065d8499fa761add95014d08a', 'request_id': 'req-d8958591-373c-41a3-919c-0fafe7a80764'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:32.358293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7214321065d8499fa761add95014d08a', 'request_id': 'req-d8958591-373c-41a3-919c-0fafe7a80764'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:32.368904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] The token cache is being invalidate because project 7214321065d8499fa761add95014d08a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:32.369187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:32.369867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d8958591-373c-41a3-919c-0fafe7a80764 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:32.371459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1048/2093] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:32 2022] DELETE /identity/v3/projects/7214321065d8499fa761add95014d08a => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:32.376734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae980d81-9f97-449e-ad9a-45962c7d6d3a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.377050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae980d81-9f97-449e-ad9a-45962c7d6d3a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.377320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae980d81-9f97-449e-ad9a-45962c7d6d3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.398904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ae980d81-9f97-449e-ad9a-45962c7d6d3a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:32.400114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ae980d81-9f97-449e-ad9a-45962c7d6d3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:32.467116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1046/2094] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:32.537437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef86994-08a2-4aaa-b7f7-de52c23c35d0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.537744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef86994-08a2-4aaa-b7f7-de52c23c35d0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.538016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eef86994-08a2-4aaa-b7f7-de52c23c35d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.561523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-eef86994-08a2-4aaa-b7f7-de52c23c35d0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:32.562832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-eef86994-08a2-4aaa-b7f7-de52c23c35d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:32.572257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1049/2095] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:32.577305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae980d81-9f97-449e-ad9a-45962c7d6d3a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:32.589470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae980d81-9f97-449e-ad9a-45962c7d6d3a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:32.596778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:32.597535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.597849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.598273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.599283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:32.599283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:32.601019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:32.615007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1047/2096] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:32.619460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef86994-08a2-4aaa-b7f7-de52c23c35d0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:32.629482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eef86994-08a2-4aaa-b7f7-de52c23c35d0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:32.634564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:32.635176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.635454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.635742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.636391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:32.638352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:32.668123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1050/2097] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:32 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:32.672276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:32.682487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4271e3-9f0f-4979-8c56-fd4d7f77673b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:32.742144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:32.742896 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.743172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.743444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.744085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:32.746052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:32.771616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1048/2098] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:32 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:32.775927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:32.786770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6b0e467-41d8-433f-8ab9-e2b9b013757a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:32.791633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:32.792231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.792508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.792780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.793425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:32.795074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:32.805154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1051/2099] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:32.809330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:32.819537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d63930-fa08-4d69-8028-87126cc0e959 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:32.824568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c7d2c9-b427-41c1-a327-2d5a40c3fa87 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:32.825155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c7d2c9-b427-41c1-a327-2d5a40c3fa87 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.825422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c7d2c9-b427-41c1-a327-2d5a40c3fa87 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.825692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c7d2c9-b427-41c1-a327-2d5a40c3fa87 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.826356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c7d2c9-b427-41c1-a327-2d5a40c3fa87 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:32.827980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2c7d2c9-b427-41c1-a327-2d5a40c3fa87 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:32.843603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1049/2100] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:32 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:32.848172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:32.865503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd877d0-2868-4d99-b31c-66e1e7a0c69c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:32.871321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:32.871980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.872253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.872519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] PATH_INFO: `/v3/projects/30c052816d5544b38124d98d05b32798/users/92d1fb3d60734f5e9bde384f96c73d30/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.888381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=30c052816d5544b38124d98d05b32798, user_id=92d1fb3d60734f5e9bde384f96c73d30, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:32.891087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:32.907602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1052/2101] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:32 2022] PUT /identity/v3/projects/30c052816d5544b38124d98d05b32798/users/92d1fb3d60734f5e9bde384f96c73d30/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:32.913889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:32.914317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:32.914683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:32.939290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:32.940592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:32.979333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:33.002756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1050/2102] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:33.012618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:33.014542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:33.055956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:33.073247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d82b209-1c31-4902-a5c8-cf3a2cde0f8b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:33.077036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a1494d7-2042-4c73-a75a-4730245b87ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:33.077639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1494d7-2042-4c73-a75a-4730245b87ad service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:33.077914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1494d7-2042-4c73-a75a-4730245b87ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:33.078219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a1494d7-2042-4c73-a75a-4730245b87ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:33.087482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1494d7-2042-4c73-a75a-4730245b87ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:33.088906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a1494d7-2042-4c73-a75a-4730245b87ad service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:33.098705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1053/2103] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:37.009106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:37.011405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:37.113748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d32b7c9-3ada-4231-a48f-706cc78df3d1 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:37.120919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:37.121572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:37.121928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:37.122200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:37.124655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:37.238731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:37.240492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:37.255939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1051/2104] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:37 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 247 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:37.793761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726cac65-f49e-4926-a66e-cdaf476d03e9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:37.794602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726cac65-f49e-4926-a66e-cdaf476d03e9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:37.795047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-726cac65-f49e-4926-a66e-cdaf476d03e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:37.830096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-726cac65-f49e-4926-a66e-cdaf476d03e9 None None] MFA Rules not processed for user `92d1fb3d60734f5e9bde384f96c73d30`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:37.831328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-726cac65-f49e-4926-a66e-cdaf476d03e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:37.899555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1054/2105] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:37 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:37.904131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:37.915864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b2ba07-ffda-47ea-b499-bacb0b3edf59 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:37.920211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:37.920872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:37.921076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:37.921392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:37.922075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:37.923980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:37.953017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1052/2106] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:37 2022] POST /identity/v3/projects => generated 385 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:37.957220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726cac65-f49e-4926-a66e-cdaf476d03e9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:37.968772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-726cac65-f49e-4926-a66e-cdaf476d03e9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:38.030156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:38.030781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.031058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.031332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.031989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:38.033959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:38.058751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1055/2107] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:37 2022] POST /identity/v3/users => generated 299 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:38.062957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:38.073610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7088b7b-6370-4881-9d6e-a1e646106e41 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:38.078163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:38.078680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.078999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.079268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.079900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:38.081488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:38.096766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1053/2108] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:38.098874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:38.110225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:38.111944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:38.112943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73649271-8a08-4e03-930b-c219db732337 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:38.118153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:38.119210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.119210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.119368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] PATH_INFO: `/v3/projects/75115cecb31d48abb6236ce3dc8cd829/users/3b65db35145749bfb82f0dc640f64a95/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.126154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cccef2-4031-4e14-ae6b-c2ed42ffffac admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:38.132753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:38.133484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.133685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.133933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] PATH_INFO: `/v3/users/ea4a2273af6a4ca48f09ca427849d66a` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.135130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ea4a2273af6a4ca48f09ca427849d66a)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:38.137044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:38.137600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75115cecb31d48abb6236ce3dc8cd829, user_id=3b65db35145749bfb82f0dc640f64a95, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:38.140497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:38.158530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1056/2109] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:38 2022] PUT /identity/v3/projects/75115cecb31d48abb6236ce3dc8cd829/users/3b65db35145749bfb82f0dc640f64a95/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:38.162505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:38.193822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ea4a2273af6a4ca48f09ca427849d66a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:38.199597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:38.201812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ea4a2273af6a4ca48f09ca427849d66a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:38.202617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ea4a2273af6a4ca48f09ca427849d66a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:38.203378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ea4a2273af6a4ca48f09ca427849d66a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:38.204631 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:38.214440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e8722ef5-74aa-400b-81b0-fc1c62a2f6b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ea4a2273af6a4ca48f09ca427849d66a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:38.230190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db3c1498-6eb4-417c-ab7b-1bcc59eb1bb9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:38.232641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1054/2110] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:38 2022] DELETE /identity/v3/users/ea4a2273af6a4ca48f09ca427849d66a => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:38.234983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:38.235698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.237092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.237368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.237633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.238317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.238649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.239506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:38.241137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:38.252397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1057/2111] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:38 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:38.256514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:38.256899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:38.258156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:38.258679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:38.273185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08489806-a38f-49f7-8d83-8402846c863c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:38.321670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1055/2112] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:38.331356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:38.332698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:38.334807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:38.335512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.335789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.336063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] PATH_INFO: `/v3/projects/75115cecb31d48abb6236ce3dc8cd829/users/3b65db35145749bfb82f0dc640f64a95/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.337964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75115cecb31d48abb6236ce3dc8cd829, user_id=3b65db35145749bfb82f0dc640f64a95, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:38.341075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:38.352659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1058/2113] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:38 2022] PUT /identity/v3/projects/75115cecb31d48abb6236ce3dc8cd829/users/3b65db35145749bfb82f0dc640f64a95/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 97 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:38.395693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec326984-8e8d-4d61-9484-e6aea74bb3b8 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:38.448163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:38.448819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:38.449116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:38.449384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:38.496912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:38.530791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:38.534134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:38.557827 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1056/2114] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:39.042115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:39.044897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:39.062637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc03049-0096-41ee-99a8-2adcc177a6d2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:39.071074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:39.071888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:39.072510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:39.072930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] PATH_INFO: `/v3/projects/cff2f494a8064952a7ff8184ef8fea7f` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:39.074914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cff2f494a8064952a7ff8184ef8fea7f)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:39.079147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:39.177541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cff2f494a8064952a7ff8184ef8fea7f', 'request_id': 'req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.188089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cff2f494a8064952a7ff8184ef8fea7f', 'request_id': 'req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.196276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cff2f494a8064952a7ff8184ef8fea7f', 'request_id': 'req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.213894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] The token cache is being invalidate because project cff2f494a8064952a7ff8184ef8fea7f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:39.214964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.216214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.221658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1059/2115] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:39 2022] DELETE /identity/v3/projects/cff2f494a8064952a7ff8184ef8fea7f => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:39.223997 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:39.234941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e62d7e-4eb7-44c1-afb4-41e3b46be7e6 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:39.314549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:39.315628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:39.316160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:39.316635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] PATH_INFO: `/v3/users/b220b25271564d4dbaaffd0be12d5a04` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:39.318132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b220b25271564d4dbaaffd0be12d5a04)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:39.321294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:39.387508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b220b25271564d4dbaaffd0be12d5a04'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.395626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b220b25271564d4dbaaffd0be12d5a04'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.396493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b220b25271564d4dbaaffd0be12d5a04'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.397356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b220b25271564d4dbaaffd0be12d5a04'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.412205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-392b502b-f8dc-4b9e-bc7b-607bd0ba7ef0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b220b25271564d4dbaaffd0be12d5a04'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:39.436978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1057/2116] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:39 2022] DELETE /identity/v3/users/b220b25271564d4dbaaffd0be12d5a04 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:39.868717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:39.870184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:39.883716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0d45440-43f4-4c1a-b8c7-265d6e926fb7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:39.932221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:39.947161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:39.948791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:39.948791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:39.948791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] PATH_INFO: `/v3/projects/0e3cda5f16b5491eaee9a2467e5d7453` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:39.949524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0e3cda5f16b5491eaee9a2467e5d7453)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:39.951927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:40.021083 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0e3cda5f16b5491eaee9a2467e5d7453', 'request_id': 'req-9a126992-5295-4b3c-aa6d-4f6c94a68d22'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:40.035700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0e3cda5f16b5491eaee9a2467e5d7453', 'request_id': 'req-9a126992-5295-4b3c-aa6d-4f6c94a68d22'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:40.044113 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0e3cda5f16b5491eaee9a2467e5d7453', 'request_id': 'req-9a126992-5295-4b3c-aa6d-4f6c94a68d22'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:40.056139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] The token cache is being invalidate because project 0e3cda5f16b5491eaee9a2467e5d7453 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:40.056426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:40.057478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9a126992-5295-4b3c-aa6d-4f6c94a68d22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:40.060182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1060/2117] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:39 2022] DELETE /identity/v3/projects/0e3cda5f16b5491eaee9a2467e5d7453 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:40.065765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a96f97-5a2f-49a7-b8b3-91aaa983f363 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.066080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a96f97-5a2f-49a7-b8b3-91aaa983f363 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.066353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a96f97-5a2f-49a7-b8b3-91aaa983f363 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.086805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-23a96f97-5a2f-49a7-b8b3-91aaa983f363 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:40.087968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-23a96f97-5a2f-49a7-b8b3-91aaa983f363 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:40.144415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:40.158179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1058/2118] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:40.166274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b6b78-8976-4846-bc22-c04b8068660a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.166689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b6b78-8976-4846-bc22-c04b8068660a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.167036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c6b6b78-8976-4846-bc22-c04b8068660a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.197922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4c6b6b78-8976-4846-bc22-c04b8068660a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:40.199784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4c6b6b78-8976-4846-bc22-c04b8068660a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:40.216266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1061/2119] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:40.217497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a96f97-5a2f-49a7-b8b3-91aaa983f363 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:40.233751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a96f97-5a2f-49a7-b8b3-91aaa983f363 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:40.243885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:40.245115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.245563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.247386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.259700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:40.260135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:40.262350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:40.280728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1059/2120] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:40.287868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6b6b78-8976-4846-bc22-c04b8068660a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:40.307365 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c6b6b78-8976-4846-bc22-c04b8068660a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:40.313556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:40.314427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.314844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.315413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.316254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:40.318784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:40.361506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1062/2121] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:40 2022] POST /identity/v3/projects => generated 383 bytes in 77 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:40.365718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:40.382150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8de019d1-d0e3-45cc-9e0c-d597dd9a4409 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:40.494466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:40.496066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.496066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.496066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.496680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:40.498992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:40.542255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1060/2122] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:40 2022] POST /identity/v3/users => generated 292 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:40.548928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:40.565700 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35eefff4-8802-4675-9f5a-5740db86db2d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:40.572419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:40.573399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.573774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.574197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.574962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:40.576796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:40.594141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1063/2123] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:40 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:40.597282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:40.617947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f21105f4-d836-4d53-9852-0ef14bfbaabf admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:40.635646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-467599cc-7be1-4c94-ab66-2b396ec33ba8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:40.636992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467599cc-7be1-4c94-ab66-2b396ec33ba8 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.637415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467599cc-7be1-4c94-ab66-2b396ec33ba8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.637559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-467599cc-7be1-4c94-ab66-2b396ec33ba8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.638930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467599cc-7be1-4c94-ab66-2b396ec33ba8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:40.641114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-467599cc-7be1-4c94-ab66-2b396ec33ba8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:40.658934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1061/2124] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:40 2022] GET /identity/v3/roles => generated 1420 bytes in 63 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:40.663482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:40.673710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a51e638-164a-45c3-9651-5f5f0d40e284 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:40.678782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:40.679446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.679726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.679987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] PATH_INFO: `/v3/projects/08cd6ab79dc44fc997a0a03bd67c370f/users/685febacfd6f40079575884935ea9a1a/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.698087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] RBAC: Authorizing `identity:create_grant(project_id=08cd6ab79dc44fc997a0a03bd67c370f, user_id=685febacfd6f40079575884935ea9a1a, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:40.719835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:40.743964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1064/2125] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:40 2022] PUT /identity/v3/projects/08cd6ab79dc44fc997a0a03bd67c370f/users/685febacfd6f40079575884935ea9a1a/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 81 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:40.757104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e97bb-0811-48ed-a9a8-46969cac60bb None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:40.757104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e97bb-0811-48ed-a9a8-46969cac60bb None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:40.757104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730e97bb-0811-48ed-a9a8-46969cac60bb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:40.813795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-730e97bb-0811-48ed-a9a8-46969cac60bb None None] MFA Rules not processed for user `685febacfd6f40079575884935ea9a1a`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:40.815040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-730e97bb-0811-48ed-a9a8-46969cac60bb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:40.949388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1062/2126] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:40 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:40.962895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:40.964789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:41.077689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff06763-469c-4a1c-b520-f75b5382fa5d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:41.089151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-307fec75-4339-4da8-b42e-e0bd329e6d4e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:41.089801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307fec75-4339-4da8-b42e-e0bd329e6d4e service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:41.090097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307fec75-4339-4da8-b42e-e0bd329e6d4e service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:41.090373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-307fec75-4339-4da8-b42e-e0bd329e6d4e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:41.111235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307fec75-4339-4da8-b42e-e0bd329e6d4e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:41.112731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-307fec75-4339-4da8-b42e-e0bd329e6d4e service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:41.132773 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1065/2127] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 170 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:41.182667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e97bb-0811-48ed-a9a8-46969cac60bb None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:41.269897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:41.297107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730e97bb-0811-48ed-a9a8-46969cac60bb None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:41.311280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:41.311950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:41.312220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:41.312486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] PATH_INFO: `/v3/users/685febacfd6f40079575884935ea9a1a` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:41.313641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] RBAC: Authorizing `identity:delete_user(user_id=685febacfd6f40079575884935ea9a1a)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:41.318040 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:41.437022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '685febacfd6f40079575884935ea9a1a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:41.447641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '685febacfd6f40079575884935ea9a1a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:41.448679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '685febacfd6f40079575884935ea9a1a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:41.449448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '685febacfd6f40079575884935ea9a1a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:41.465553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '685febacfd6f40079575884935ea9a1a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:41.489215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1063/2128] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:41 2022] DELETE /identity/v3/users/685febacfd6f40079575884935ea9a1a => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:41.494279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:41.494688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:41.495203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:41.525538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:41.527475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:41.559823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:41.641295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1066/2129] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:41.661489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:41.664057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:41.755245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0e1ec48-563f-4fff-b6e4-f7fa80a1588b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:41.762872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41905803-b310-4ea2-9087-72af3176fd62 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:41.763517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41905803-b310-4ea2-9087-72af3176fd62 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:41.763891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41905803-b310-4ea2-9087-72af3176fd62 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:41.764060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41905803-b310-4ea2-9087-72af3176fd62 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:41.785439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41905803-b310-4ea2-9087-72af3176fd62 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:41.787007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41905803-b310-4ea2-9087-72af3176fd62 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:41.800357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1064/2130] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:42.443243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:42.445426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:42.463681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd9126-f0d0-4b04-9eb2-0088922e2e19 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:42.486154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:42.487670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:42.488600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:42.489382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] PATH_INFO: `/v3/projects/08cd6ab79dc44fc997a0a03bd67c370f` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:42.493511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08cd6ab79dc44fc997a0a03bd67c370f)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:42.496000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:42.616911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08cd6ab79dc44fc997a0a03bd67c370f', 'request_id': 'req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:42.628894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08cd6ab79dc44fc997a0a03bd67c370f', 'request_id': 'req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:42.644431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08cd6ab79dc44fc997a0a03bd67c370f', 'request_id': 'req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:42.668894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] The token cache is being invalidate because project 08cd6ab79dc44fc997a0a03bd67c370f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:42.669197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:42.670384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0d11d6fe-2073-45cc-9c30-76b1a4dd6123 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:42.672412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1067/2131] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:42 2022] DELETE /identity/v3/projects/08cd6ab79dc44fc997a0a03bd67c370f => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:42.678596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf18f07-8dc0-412f-832f-18c155350a6f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:42.679202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf18f07-8dc0-412f-832f-18c155350a6f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:42.679625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf18f07-8dc0-412f-832f-18c155350a6f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:42.717071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3cf18f07-8dc0-412f-832f-18c155350a6f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:42.719191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3cf18f07-8dc0-412f-832f-18c155350a6f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:42.794780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1065/2132] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:42.803950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de62c4-c982-4fdf-861e-f8d331ccf065 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:42.804235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de62c4-c982-4fdf-861e-f8d331ccf065 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:42.804649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9de62c4-c982-4fdf-861e-f8d331ccf065 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:42.829388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b9de62c4-c982-4fdf-861e-f8d331ccf065 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:42.830690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b9de62c4-c982-4fdf-861e-f8d331ccf065 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:42.842089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1068/2133] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:42.844983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf18f07-8dc0-412f-832f-18c155350a6f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:42.856748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf18f07-8dc0-412f-832f-18c155350a6f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:42.862206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:42.862839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:42.863112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:42.863380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:42.864119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:42.864603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:42.866282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:42.879362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1066/2134] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:42.883909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9de62c4-c982-4fdf-861e-f8d331ccf065 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:42.895888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9de62c4-c982-4fdf-861e-f8d331ccf065 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:42.902332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:42.902949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:42.903222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:42.903492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:42.904122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:42.906028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:42.954453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1069/2135] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:42 2022] POST /identity/v3/projects => generated 373 bytes in 71 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:42.957080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:42.965614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:42.971527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395ffe1b-d8d6-4bd2-8dad-3d5525f0ad6c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.126168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.136802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.137104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.137379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.138092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.140070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.197113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1067/2136] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:42 2022] POST /identity/v3/users => generated 287 bytes in 238 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:43.198750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.214040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f66e3268-baca-4ca6-862c-fb67ba3c9a47 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.225526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.226122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.226403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.226673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.227368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.229340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.243384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1070/2137] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:43 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:43.247265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.264609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c138c31a-7968-49c6-b4aa-68518aac8405 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.275936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c649b98-7c18-4aeb-ae79-62c87003ded8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.276551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c649b98-7c18-4aeb-ae79-62c87003ded8 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.277045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c649b98-7c18-4aeb-ae79-62c87003ded8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.277610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c649b98-7c18-4aeb-ae79-62c87003ded8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.278806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c649b98-7c18-4aeb-ae79-62c87003ded8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.279906 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c649b98-7c18-4aeb-ae79-62c87003ded8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.295109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1068/2138] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:43 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:43.299216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.310329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ed157b5-89e1-4e3d-8af7-2db03807d726 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.315259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.315923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.316241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.316502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] PATH_INFO: `/v3/projects/937e7878ec7a4a4d92f457a2a94001fd/users/4a3bdc0e25d74f03b5351012fff1e10c/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.332962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] RBAC: Authorizing `identity:create_grant(project_id=937e7878ec7a4a4d92f457a2a94001fd, user_id=4a3bdc0e25d74f03b5351012fff1e10c, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.335834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.353609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1071/2139] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:43 2022] PUT /identity/v3/projects/937e7878ec7a4a4d92f457a2a94001fd/users/4a3bdc0e25d74f03b5351012fff1e10c/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:43.364501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c223bc3-c37c-434e-922f-9c5a95cc1796 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.364811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c223bc3-c37c-434e-922f-9c5a95cc1796 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.365097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c223bc3-c37c-434e-922f-9c5a95cc1796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.402579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-4c223bc3-c37c-434e-922f-9c5a95cc1796 None None] MFA Rules not processed for user `4a3bdc0e25d74f03b5351012fff1e10c`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:43.403544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4c223bc3-c37c-434e-922f-9c5a95cc1796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:43.490482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1069/2140] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:43.494462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.574463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:43.600236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0608085f-47de-4f25-ad68-e61a5685e587 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.608195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.609580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.609915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.610224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.610916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.612895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.650465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1072/2141] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/projects => generated 373 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:43.654946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c223bc3-c37c-434e-922f-9c5a95cc1796 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.667973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c223bc3-c37c-434e-922f-9c5a95cc1796 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.811420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.812210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.812574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.813084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.813928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.818370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.855557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:43.862217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1070/2142] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:43 2022] POST /identity/v3/users => generated 293 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:43.867399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.884062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-693e27b3-0d04-4067-9ed5-310993611357 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.890322 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.891378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.892229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.892712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.893830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:43.896603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:43.909239 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1073/2143] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:43 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:43.915346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:43.934838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbcf824-6d9e-49dd-804a-a4cb6389bc0b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:43.949197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3afa172f-b13b-4b94-b5e8-20d11afb22b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:43.949808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afa172f-b13b-4b94-b5e8-20d11afb22b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:43.950559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afa172f-b13b-4b94-b5e8-20d11afb22b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:43.950664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3afa172f-b13b-4b94-b5e8-20d11afb22b9 admin admin] PATH_INFO: `/v3/projects/5508b1e6694a46a3b11265fef4f2c499/users/4a487163797c4addadf55b38fd9bbbe8/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:43.987049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afa172f-b13b-4b94-b5e8-20d11afb22b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5508b1e6694a46a3b11265fef4f2c499, user_id=4a487163797c4addadf55b38fd9bbbe8, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.005035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3afa172f-b13b-4b94-b5e8-20d11afb22b9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.036850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1071/2144] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:43 2022] PUT /identity/v3/projects/5508b1e6694a46a3b11265fef4f2c499/users/4a487163797c4addadf55b38fd9bbbe8/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:44.038258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.049233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4799cdee-e06f-4c33-b9cc-0282b7e6170e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:44.083676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.083949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.084231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.128687 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] MFA Rules not processed for user `3b65db35145749bfb82f0dc640f64a95`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:44.130194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.148636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:44.149934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.150252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.150526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.151187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.153010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.168276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1074/2145] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:44 2022] GET /identity/v3/roles => generated 1420 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:44.169185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.181307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cfd6af-877a-4b8a-9325-64dbd2d67b1c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:44.188087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf95ec0-ba8e-4aa4-b95a-5c54369a9c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:44.188851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf95ec0-ba8e-4aa4-b95a-5c54369a9c9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.189458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf95ec0-ba8e-4aa4-b95a-5c54369a9c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.189737 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf95ec0-ba8e-4aa4-b95a-5c54369a9c9a admin admin] PATH_INFO: `/v3/projects/5508b1e6694a46a3b11265fef4f2c499/users/4a487163797c4addadf55b38fd9bbbe8/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.192257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf95ec0-ba8e-4aa4-b95a-5c54369a9c9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=5508b1e6694a46a3b11265fef4f2c499, user_id=4a487163797c4addadf55b38fd9bbbe8, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.195696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cf95ec0-ba8e-4aa4-b95a-5c54369a9c9a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.208152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1075/2146] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:44 2022] PUT /identity/v3/projects/5508b1e6694a46a3b11265fef4f2c499/users/4a487163797c4addadf55b38fd9bbbe8/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:44.218225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.218512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.218791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.223320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1072/2147] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:44 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:44.235144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.236639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.257530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] MFA Rules not processed for user `4a487163797c4addadf55b38fd9bbbe8`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:44.259245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.304287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b959223d-ad45-4d55-9003-c55cdeacf6fa None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:44.339723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1076/2148] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:44 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:44.352559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.354074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.366036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eda7ce6-97f6-4190-9159-753a57e8d308 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:44.367009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda7ce6-97f6-4190-9159-753a57e8d308 service glance] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.367513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda7ce6-97f6-4190-9159-753a57e8d308 service glance] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.367651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eda7ce6-97f6-4190-9159-753a57e8d308 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.372705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4eda7ce6-97f6-4190-9159-753a57e8d308 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.390232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:44.435071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:44.463341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71a03ac-bfdf-4390-94db-fd7e4fa1da80 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:44.468903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:44.469774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.470362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.471935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.481199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda7ce6-97f6-4190-9159-753a57e8d308 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.482851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eda7ce6-97f6-4190-9159-753a57e8d308 service glance] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.495396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1073/2149] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:44 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:44.576666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:44.578204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:44.618665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1077/2150] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:44 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 267 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:14:44.717911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135e399c-84c9-4501-87e7-b066033cbec3 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:44.718315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135e399c-84c9-4501-87e7-b066033cbec3 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:44.718612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135e399c-84c9-4501-87e7-b066033cbec3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:44.748842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-135e399c-84c9-4501-87e7-b066033cbec3 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:44.750119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-135e399c-84c9-4501-87e7-b066033cbec3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:44.831506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1074/2151] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:14:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:44.889901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:44.892154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:45.025709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f052af8-feb2-4a50-a718-b855d83b8cd2 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:45.051318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fadc082-0ed7-4db8-a393-ebe404c13593 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:45.051969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fadc082-0ed7-4db8-a393-ebe404c13593 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:45.052258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fadc082-0ed7-4db8-a393-ebe404c13593 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:45.052520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fadc082-0ed7-4db8-a393-ebe404c13593 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:45.090955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fadc082-0ed7-4db8-a393-ebe404c13593 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:45.092606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fadc082-0ed7-4db8-a393-ebe404c13593 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:45.105219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1078/2152] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:14:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:14:52.833712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135e399c-84c9-4501-87e7-b066033cbec3 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:52.910748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135e399c-84c9-4501-87e7-b066033cbec3 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:52.918473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:52.919272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:52.919648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:52.919972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] PATH_INFO: `/v3/users/4a3bdc0e25d74f03b5351012fff1e10c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:52.921310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a3bdc0e25d74f03b5351012fff1e10c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:52.923399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:52.984899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a3bdc0e25d74f03b5351012fff1e10c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:52.993987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a3bdc0e25d74f03b5351012fff1e10c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:52.994722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a3bdc0e25d74f03b5351012fff1e10c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:52.995316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a3bdc0e25d74f03b5351012fff1e10c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.006823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a3bdc0e25d74f03b5351012fff1e10c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.019448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:53.030646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1075/2153] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:52 2022] DELETE /identity/v3/users/4a3bdc0e25d74f03b5351012fff1e10c => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:53.035946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:53.036509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:53.036734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:53.058047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:53.059726 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:53.143116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1079/2154] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:53.151447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:53.152933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:53.231774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b23b0d-2685-4422-a913-0d1eb179ca32 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:53.235679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:53.236391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:53.236727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:53.237060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:53.249499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:53.251300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:53.261717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1076/2155] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:53.599615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:53.599615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:53.615714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:53.623772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d20c5653-22ab-497c-87b1-d0a47971ec13 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:53.633919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:53.635494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:53.635835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:53.636389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] PATH_INFO: `/v3/projects/937e7878ec7a4a4d92f457a2a94001fd` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:53.637755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] RBAC: Authorizing `identity:delete_project(project_id=937e7878ec7a4a4d92f457a2a94001fd)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:53.640252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:53.706897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '937e7878ec7a4a4d92f457a2a94001fd', 'request_id': 'req-47ed1896-9fa4-44d3-8b84-8fa3243a69da'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.722812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '937e7878ec7a4a4d92f457a2a94001fd', 'request_id': 'req-47ed1896-9fa4-44d3-8b84-8fa3243a69da'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.732431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '937e7878ec7a4a4d92f457a2a94001fd', 'request_id': 'req-47ed1896-9fa4-44d3-8b84-8fa3243a69da'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.746456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] The token cache is being invalidate because project 937e7878ec7a4a4d92f457a2a94001fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:53.747054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.747900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.749988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1080/2156] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:53 2022] DELETE /identity/v3/projects/937e7878ec7a4a4d92f457a2a94001fd => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:53.753778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:53.765685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6b01a6-805b-4b0f-82f3-de0b5dd381ff service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:53.837009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:53.837746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:53.837972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:53.838267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] PATH_INFO: `/v3/users/4a487163797c4addadf55b38fd9bbbe8` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:53.840117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] RBAC: Authorizing `identity:delete_user(user_id=4a487163797c4addadf55b38fd9bbbe8)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:53.842196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:53.907612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4a487163797c4addadf55b38fd9bbbe8'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.915097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4a487163797c4addadf55b38fd9bbbe8'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.915851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4a487163797c4addadf55b38fd9bbbe8'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.916450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4a487163797c4addadf55b38fd9bbbe8'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.930092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-853f1d3c-4998-4bd5-a637-aa70037c854c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4a487163797c4addadf55b38fd9bbbe8'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:53.963850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1077/2157] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:14:53 2022] DELETE /identity/v3/users/4a487163797c4addadf55b38fd9bbbe8 => generated 0 bytes in 210 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:54.344766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.346325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:54.367367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ed1896-9fa4-44d3-8b84-8fa3243a69da admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.440680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.441227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.441497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.441765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] PATH_INFO: `/v3/projects/5508b1e6694a46a3b11265fef4f2c499` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.443128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5508b1e6694a46a3b11265fef4f2c499)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.445059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:54.500393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5508b1e6694a46a3b11265fef4f2c499', 'request_id': 'req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:54.509947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5508b1e6694a46a3b11265fef4f2c499', 'request_id': 'req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:54.516941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5508b1e6694a46a3b11265fef4f2c499', 'request_id': 'req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:54.528010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] The token cache is being invalidate because project 5508b1e6694a46a3b11265fef4f2c499 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:14:54.528482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:54.529229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be28a8fd-7caf-4e59-99d4-62a28bbb8b8d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:14:54.530838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1081/2158] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:14:54 2022] DELETE /identity/v3/projects/5508b1e6694a46a3b11265fef4f2c499 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:14:54.536100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3628db2-3093-4a55-af66-9a687b3e1c34 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.536504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3628db2-3093-4a55-af66-9a687b3e1c34 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.536780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3628db2-3093-4a55-af66-9a687b3e1c34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.557507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c3628db2-3093-4a55-af66-9a687b3e1c34 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:54.558758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c3628db2-3093-4a55-af66-9a687b3e1c34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:54.572657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:54.631669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1078/2159] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:54.640778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c47af9-2406-44b1-94cc-45fb98a41ccf None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.641020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c47af9-2406-44b1-94cc-45fb98a41ccf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.641335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44c47af9-2406-44b1-94cc-45fb98a41ccf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.669709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-44c47af9-2406-44b1-94cc-45fb98a41ccf None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:54.671078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-44c47af9-2406-44b1-94cc-45fb98a41ccf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:54.680639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1082/2160] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:54.684928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3628db2-3093-4a55-af66-9a687b3e1c34 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.694578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3628db2-3093-4a55-af66-9a687b3e1c34 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.699682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.700256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.700738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.700874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.701573 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:14:54.701875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.703534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:54.715640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1079/2161] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:14:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:14:54.720182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c47af9-2406-44b1-94cc-45fb98a41ccf None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.729778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44c47af9-2406-44b1-94cc-45fb98a41ccf None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.736188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41929891-1419-4f25-b4b7-09ab76eff1d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.736788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41929891-1419-4f25-b4b7-09ab76eff1d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.737056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41929891-1419-4f25-b4b7-09ab76eff1d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.737320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41929891-1419-4f25-b4b7-09ab76eff1d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.737979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41929891-1419-4f25-b4b7-09ab76eff1d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.740297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41929891-1419-4f25-b4b7-09ab76eff1d0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:54.750458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:54.775192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1083/2162] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:54 2022] POST /identity/v3/projects => generated 377 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:54.776327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.786936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28baa4d6-b391-4501-a00f-caf77f3834a7 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.844865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812f8f9-45ca-4bdd-a9e1-4ae7d39a96d8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.845148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812f8f9-45ca-4bdd-a9e1-4ae7d39a96d8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.845419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a812f8f9-45ca-4bdd-a9e1-4ae7d39a96d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.848834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.849450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.849727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.850089 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.850726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.852694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:54.867848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a812f8f9-45ca-4bdd-a9e1-4ae7d39a96d8 None None] MFA Rules not processed for user `92d1fb3d60734f5e9bde384f96c73d30`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:54.869148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a812f8f9-45ca-4bdd-a9e1-4ae7d39a96d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:54.877047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1080/2163] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:54 2022] POST /identity/v3/users => generated 289 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:54.881318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.891615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b17774-487c-49b7-8cee-51b0bcc76035 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.896439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.897129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.897316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.897548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.898264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.899847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:54.912415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1081/2164] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:54.916217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.926341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fde04e-d144-4b86-af62-4a7670f3a679 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.929409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1084/2165] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:54 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:14:54.931538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-632a54f4-e97a-4b1e-9634-7374d7076aa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.932127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632a54f4-e97a-4b1e-9634-7374d7076aa8 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.932407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632a54f4-e97a-4b1e-9634-7374d7076aa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.932674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-632a54f4-e97a-4b1e-9634-7374d7076aa8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.933397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632a54f4-e97a-4b1e-9634-7374d7076aa8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.935024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-632a54f4-e97a-4b1e-9634-7374d7076aa8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:54.946354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1082/2166] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:14:54.950341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a812f8f9-45ca-4bdd-a9e1-4ae7d39a96d8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:54.961189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a812f8f9-45ca-4bdd-a9e1-4ae7d39a96d8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:54.966071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:54.966816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:54.967139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:54.967419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] PATH_INFO: `/v3/projects/c8d38aa62ca24ff1a4a993be430aff74/users/d139e8970cc64c679f5ef4ec54867219/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:54.985104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] RBAC: Authorizing `identity:create_grant(project_id=c8d38aa62ca24ff1a4a993be430aff74, user_id=d139e8970cc64c679f5ef4ec54867219, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:54.987911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.004040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1085/2167] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:54 2022] PUT /identity/v3/projects/c8d38aa62ca24ff1a4a993be430aff74/users/d139e8970cc64c679f5ef4ec54867219/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:55.015073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0e422d-2974-4664-8c4f-401d2de200fd None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.016717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0e422d-2974-4664-8c4f-401d2de200fd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.017009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0e422d-2974-4664-8c4f-401d2de200fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.047027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-fd0e422d-2974-4664-8c4f-401d2de200fd None None] MFA Rules not processed for user `d139e8970cc64c679f5ef4ec54867219`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:55.048265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-fd0e422d-2974-4664-8c4f-401d2de200fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:55.115529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1083/2168] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:55 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:55.120089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.195589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a72e19-c031-4009-879f-fe6cd4b8b0da admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.201152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.201712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.201965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.202285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.202926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.204881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.246459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1086/2169] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:55 2022] POST /identity/v3/projects => generated 375 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:55.248162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0e422d-2974-4664-8c4f-401d2de200fd None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.254823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:55.264107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0e422d-2974-4664-8c4f-401d2de200fd None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.336996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.337608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.337883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.338190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.338835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.340782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.368206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1084/2170] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:55 2022] POST /identity/v3/users => generated 288 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:55.369597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.380769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b35316-2f5d-4fbd-8393-32f1d8d5bbfa admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.386559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.387242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.387578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.387909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.388647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.390304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.401669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1087/2171] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:55 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:55.405893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.416695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22e353e-12db-4a23-b049-e18e524345cc admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.422121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c1e308-4dfc-47e7-a8e2-8669b4970a9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.422802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c1e308-4dfc-47e7-a8e2-8669b4970a9a admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.422980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c1e308-4dfc-47e7-a8e2-8669b4970a9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.423249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c1e308-4dfc-47e7-a8e2-8669b4970a9a admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.423914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c1e308-4dfc-47e7-a8e2-8669b4970a9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.425516 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c1e308-4dfc-47e7-a8e2-8669b4970a9a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.436071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1085/2172] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:55.440382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.451031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70202536-3a4a-42db-9dfc-83f5301a2e52 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.456489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.457079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.457350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.457608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] PATH_INFO: `/v3/projects/f480210acbc6449b9835ce9631484c21/users/3f47669a2923420492a03045a650c5cc/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.474851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f480210acbc6449b9835ce9631484c21, user_id=3f47669a2923420492a03045a650c5cc, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.476950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.491985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1088/2173] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:55 2022] PUT /identity/v3/projects/f480210acbc6449b9835ce9631484c21/users/3f47669a2923420492a03045a650c5cc/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:55.503306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702c3e3-27bc-42eb-82a5-22cc86f8f923 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.503588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702c3e3-27bc-42eb-82a5-22cc86f8f923 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.503859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3702c3e3-27bc-42eb-82a5-22cc86f8f923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.534286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3702c3e3-27bc-42eb-82a5-22cc86f8f923 None None] MFA Rules not processed for user `3f47669a2923420492a03045a650c5cc`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:55.538134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3702c3e3-27bc-42eb-82a5-22cc86f8f923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:55.611259 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1086/2174] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:55 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:55.615759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.644760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:55.688763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a21b2b8a-3e8f-4924-a436-5d77b2272c86 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.693480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.694117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.694390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.694659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.695300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.697200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.738092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1089/2175] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:14:55 2022] POST /identity/v3/projects => generated 377 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:14:55.739933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702c3e3-27bc-42eb-82a5-22cc86f8f923 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.750707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3702c3e3-27bc-42eb-82a5-22cc86f8f923 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.809133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.809748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.810068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.810345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.810994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.812924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.830728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:55.836588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1087/2176] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:14:55 2022] POST /identity/v3/users => generated 295 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:14:55.841330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.854829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-853c9f73-6888-49ef-811a-ce8ec8a238f4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.861036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.861662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.862109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.862426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.863210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.865363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.878260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1090/2177] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:55 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:55.882830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.895394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91afcfcd-45bd-46ac-b1d7-720adefc9656 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:55.900972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:55.901660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:55.901934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:55.902323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] PATH_INFO: `/v3/projects/6d21a3f8e0cf4ef1850e2488ec29cb36/users/3bfef25ab2c04d99bf750639a5ce2bed/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:55.920701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d21a3f8e0cf4ef1850e2488ec29cb36, user_id=3bfef25ab2c04d99bf750639a5ce2bed, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:55.923544 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:55.939604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1088/2178] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:55 2022] PUT /identity/v3/projects/6d21a3f8e0cf4ef1850e2488ec29cb36/users/3bfef25ab2c04d99bf750639a5ce2bed/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:55.944367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:55.954901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86a90017-6429-4eac-93b7-e6540ad28f46 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:56.016377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7af9ffa9-6b66-4351-8e56-b7883e3356c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:56.016994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9ffa9-6b66-4351-8e56-b7883e3356c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:56.017271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9ffa9-6b66-4351-8e56-b7883e3356c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:56.017541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7af9ffa9-6b66-4351-8e56-b7883e3356c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:56.018209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af9ffa9-6b66-4351-8e56-b7883e3356c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:56.019846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7af9ffa9-6b66-4351-8e56-b7883e3356c4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:56.029699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1091/2179] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:14:55 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:14:56.034062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:56.044697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf190f79-7088-4db4-a288-60a491ba68ba admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:56.050428 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:56.051079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:56.051373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:56.051767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] PATH_INFO: `/v3/projects/6d21a3f8e0cf4ef1850e2488ec29cb36/users/3bfef25ab2c04d99bf750639a5ce2bed/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:56.053940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6d21a3f8e0cf4ef1850e2488ec29cb36, user_id=3bfef25ab2c04d99bf750639a5ce2bed, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:56.056649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:56.067446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1089/2180] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:14:56 2022] PUT /identity/v3/projects/6d21a3f8e0cf4ef1850e2488ec29cb36/users/3bfef25ab2c04d99bf750639a5ce2bed/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:14:56.078261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8084a-bbd6-4926-b6c4-052264129583 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:56.078539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8084a-bbd6-4926-b6c4-052264129583 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:56.078808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c8084a-bbd6-4926-b6c4-052264129583 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:56.109003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f1c8084a-bbd6-4926-b6c4-052264129583 None None] MFA Rules not processed for user `3bfef25ab2c04d99bf750639a5ce2bed`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:56.110245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f1c8084a-bbd6-4926-b6c4-052264129583 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:56.180720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1092/2181] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:14:56 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:56.192972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:56.195009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:56.263287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b850d1-e205-42ee-bd95-9bccf62b6cc1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:56.266966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:56.267590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:56.267892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:56.268191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:56.329841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:56.331344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:56.340736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1090/2182] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:14:56 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:14:59.514665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:59.515170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:59.515269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:59.539286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:14:59.540598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:59.554391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:14:59.602815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1093/2183] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:14:59 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:14:59.615888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:14:59.617309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:14:59.693786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf190b82-6220-437a-9fe8-d21ce609075d service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:14:59.701208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:14:59.701836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:14:59.702154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:14:59.702423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:14:59.712348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:14:59.714240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:14:59.723345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1091/2184] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:14:59 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 29 02:15:00.268467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:00.269985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:00.294178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80053f46-7789-45ba-9249-a07a4d715133 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:00.301373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:00.302456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:00.303049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:00.303559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:00.398079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:00.400448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:00.413287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1094/2185] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:01.012658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:01.023031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-456d7b14-ca6f-4dd5-ac0a-71250868b06c service swift] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:01.027223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d635c4-451b-40f5-b773-65a54919c4ea service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:01.027835 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d635c4-451b-40f5-b773-65a54919c4ea service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:01.028111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d635c4-451b-40f5-b773-65a54919c4ea service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:01.028380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d635c4-451b-40f5-b773-65a54919c4ea service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:01.035146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:01.039793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d635c4-451b-40f5-b773-65a54919c4ea service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:01.041417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d635c4-451b-40f5-b773-65a54919c4ea service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:01.052844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1092/2186] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:01 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:10.407909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=46 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:11.053885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:13.740174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:13.809269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b3551e6-f00a-480d-b96d-8e96475da0c7 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:13.814097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:13.814854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:13.815117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:13.815407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] PATH_INFO: `/v3/users/d139e8970cc64c679f5ef4ec54867219` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:13.816539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d139e8970cc64c679f5ef4ec54867219)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:13.818519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:13.880579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd139e8970cc64c679f5ef4ec54867219'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:13.889534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd139e8970cc64c679f5ef4ec54867219'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:13.890583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd139e8970cc64c679f5ef4ec54867219'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:13.891387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd139e8970cc64c679f5ef4ec54867219'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:13.901500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd139e8970cc64c679f5ef4ec54867219'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:13.919745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1095/2187] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:13 2022] DELETE /identity/v3/users/d139e8970cc64c679f5ef4ec54867219 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:13.924464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:13.924985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:13.925088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:13.943923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:13.945222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:14.005633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1093/2188] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:14.015712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:14.017164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:14.078539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f009a1c9-1138-4038-b8d9-620f58803c70 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:14.083056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:14.083657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:14.083927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:14.084168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:14.094453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:14.095815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:14.106056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1096/2189] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:14.397030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:14.398532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:14.411338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a44f95b-a111-4c68-8114-e6fc4eef7349 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:14.416357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:14.417142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:14.417561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:14.417978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] PATH_INFO: `/v3/projects/c8d38aa62ca24ff1a4a993be430aff74` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:14.419250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c8d38aa62ca24ff1a4a993be430aff74)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:14.421274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:14.472346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c8d38aa62ca24ff1a4a993be430aff74', 'request_id': 'req-9af846be-e2ef-42f4-a96a-acb3652f2405'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.481831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c8d38aa62ca24ff1a4a993be430aff74', 'request_id': 'req-9af846be-e2ef-42f4-a96a-acb3652f2405'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.488242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c8d38aa62ca24ff1a4a993be430aff74', 'request_id': 'req-9af846be-e2ef-42f4-a96a-acb3652f2405'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.498890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] The token cache is being invalidate because project c8d38aa62ca24ff1a4a993be430aff74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:14.499172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.499912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.501517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1094/2190] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:14 2022] DELETE /identity/v3/projects/c8d38aa62ca24ff1a4a993be430aff74 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:14.505785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:14.517205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab2d758c-9757-4950-8d12-de9aee98ab54 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:14.535636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:14.576134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:14.576777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:14.577055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:14.577327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] PATH_INFO: `/v3/users/3f47669a2923420492a03045a650c5cc` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:14.578627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3f47669a2923420492a03045a650c5cc)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:14.580598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:14.630618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3f47669a2923420492a03045a650c5cc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.638288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3f47669a2923420492a03045a650c5cc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.639077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3f47669a2923420492a03045a650c5cc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.639840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3f47669a2923420492a03045a650c5cc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.649222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6e3612b1-816e-4a11-807d-0487a3987ba1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3f47669a2923420492a03045a650c5cc'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:14.668485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1097/2191] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:14 2022] DELETE /identity/v3/users/3f47669a2923420492a03045a650c5cc => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:14.936239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:14.938006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:14.960898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9af846be-e2ef-42f4-a96a-acb3652f2405 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:15.036340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:15.036991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:15.038384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:15.038384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] PATH_INFO: `/v3/users/265039311b86450ba4c3addc6f17c505` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:15.040628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] RBAC: Authorizing `identity:delete_user(user_id=265039311b86450ba4c3addc6f17c505)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:15.043228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:15.075534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:15.098296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '265039311b86450ba4c3addc6f17c505'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.105775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '265039311b86450ba4c3addc6f17c505'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.106903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '265039311b86450ba4c3addc6f17c505'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.107021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '265039311b86450ba4c3addc6f17c505'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.124878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '265039311b86450ba4c3addc6f17c505'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.144907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1095/2192] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:14 2022] DELETE /identity/v3/users/265039311b86450ba4c3addc6f17c505 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:15.151826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:15.152276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:15.152398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:15.172414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:15.173633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:15.237274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1098/2193] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:15.255430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:15.257511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:15.296833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:15.298935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:15.314297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277cda81-6c40-4a4c-882d-4f4365fc3de8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:15.319884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:15.320607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:15.320947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:15.321278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] PATH_INFO: `/v3/projects/f480210acbc6449b9835ce9631484c21` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:15.322506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f480210acbc6449b9835ce9631484c21)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:15.324714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:15.350937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1c1cdc-9331-44d3-a22d-9d7a61947a03 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:15.356204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:15.356980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:15.357327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:15.357671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:15.371053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:15.372993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:15.378208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:15.386050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1096/2194] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:15.393562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f480210acbc6449b9835ce9631484c21', 'request_id': 'req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.404212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f480210acbc6449b9835ce9631484c21', 'request_id': 'req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.412115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f480210acbc6449b9835ce9631484c21', 'request_id': 'req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.426276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] The token cache is being invalidate because project f480210acbc6449b9835ce9631484c21 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:15.426648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.427382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.429195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1099/2195] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:15 2022] DELETE /identity/v3/projects/f480210acbc6449b9835ce9631484c21 => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:15.434061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:15.447663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac58b4cb-3ce5-4fe9-86ce-456a3ef632d5 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:15.520535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:15.521504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:15.521757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:15.522092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] PATH_INFO: `/v3/users/3bfef25ab2c04d99bf750639a5ce2bed` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:15.523928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] RBAC: Authorizing `identity:delete_user(user_id=3bfef25ab2c04d99bf750639a5ce2bed)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:15.526376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:15.602961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3bfef25ab2c04d99bf750639a5ce2bed'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.605027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:15.616430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3bfef25ab2c04d99bf750639a5ce2bed'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.617564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3bfef25ab2c04d99bf750639a5ce2bed'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.618334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3bfef25ab2c04d99bf750639a5ce2bed'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.630504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3bfef25ab2c04d99bf750639a5ce2bed'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.651782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1097/2196] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:15 2022] DELETE /identity/v3/users/3bfef25ab2c04d99bf750639a5ce2bed => generated 0 bytes in 218 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:15:15.689132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:15.690979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:15.705590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2edd5c7e-4ee7-4fde-a5ec-fa8d652d8c06 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:15.778010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:15.778809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:15.779121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:15.779465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] PATH_INFO: `/v3/projects/6a7d698878b044c6b458b92d61b7c7a3` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:15.780894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6a7d698878b044c6b458b92d61b7c7a3)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:15.783060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:15.845361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6a7d698878b044c6b458b92d61b7c7a3', 'request_id': 'req-e40c97d3-cf7e-425a-8ac5-a174f6304754'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.854715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6a7d698878b044c6b458b92d61b7c7a3', 'request_id': 'req-e40c97d3-cf7e-425a-8ac5-a174f6304754'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.862162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6a7d698878b044c6b458b92d61b7c7a3', 'request_id': 'req-e40c97d3-cf7e-425a-8ac5-a174f6304754'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.872698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] The token cache is being invalidate because project 6a7d698878b044c6b458b92d61b7c7a3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:15.872986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.873690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:15.875227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1100/2197] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:15 2022] DELETE /identity/v3/projects/6a7d698878b044c6b458b92d61b7c7a3 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:15.879304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:15.881192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:15.896028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7e975c-14a1-4956-a7dd-60aea912a16a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:15.957773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:15.958587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:15.958962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:15.959364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] PATH_INFO: `/v3/users/60b0177f26104516bbd60942954a7b65` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:15.960612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] RBAC: Authorizing `identity:delete_user(user_id=60b0177f26104516bbd60942954a7b65)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:15.962696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.013906 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60b0177f26104516bbd60942954a7b65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.022376 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60b0177f26104516bbd60942954a7b65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.023126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60b0177f26104516bbd60942954a7b65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.023775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60b0177f26104516bbd60942954a7b65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.038499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4e2063f3-86e5-4585-835f-ba09f7c1b642 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60b0177f26104516bbd60942954a7b65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.057074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1098/2198] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:15 2022] DELETE /identity/v3/users/60b0177f26104516bbd60942954a7b65 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:16.102863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.104640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.118406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e40c97d3-cf7e-425a-8ac5-a174f6304754 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.174372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:16.182840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.183490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.183792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.184076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] PATH_INFO: `/v3/projects/6d21a3f8e0cf4ef1850e2488ec29cb36` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.185424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6d21a3f8e0cf4ef1850e2488ec29cb36)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.188103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.242202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6d21a3f8e0cf4ef1850e2488ec29cb36', 'request_id': 'req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.251962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6d21a3f8e0cf4ef1850e2488ec29cb36', 'request_id': 'req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.258761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6d21a3f8e0cf4ef1850e2488ec29cb36', 'request_id': 'req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.270683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] The token cache is being invalidate because project 6d21a3f8e0cf4ef1850e2488ec29cb36 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:16.270972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.271627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-621c89d4-1c6f-4c2a-97a6-c1f6d16e72c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.273180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1101/2199] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:16 2022] DELETE /identity/v3/projects/6d21a3f8e0cf4ef1850e2488ec29cb36 => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:16.278535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bab6764-9f02-4adc-b13a-1311664c0244 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.278786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bab6764-9f02-4adc-b13a-1311664c0244 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.279626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bab6764-9f02-4adc-b13a-1311664c0244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.304453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0bab6764-9f02-4adc-b13a-1311664c0244 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:16.306266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0bab6764-9f02-4adc-b13a-1311664c0244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.353179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=85,keystone:INSERT=2,keystone:DELETE=13 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:16.374799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1099/2200] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:16.383129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.383409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.383681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.404552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:16.405806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.414414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1102/2201] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:16.418999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bab6764-9f02-4adc-b13a-1311664c0244 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.429323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bab6764-9f02-4adc-b13a-1311664c0244 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.434260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.434866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.435249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.435536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.436217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:15:16.436513 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.438157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.443638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.445296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.451053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1100/2202] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:15:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:15:16.455397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.465823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954612d6-66ba-476f-8140-1161f02b7c8b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.466097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b2203e5-d267-400a-aa0c-fa689feaca83 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.472465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.473147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.473476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.473973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] PATH_INFO: `/v3/projects/a381d1a27fa04724b00988146ffe0d41` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.474741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.475223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a381d1a27fa04724b00988146ffe0d41)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.475523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.475799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.476068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.476712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.477152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.478646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.510962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1101/2203] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/projects => generated 369 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:16.515272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.527635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a381d1a27fa04724b00988146ffe0d41', 'request_id': 'req-c43878fd-81c8-41fa-b646-d83af6ec8fa5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.537977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a381d1a27fa04724b00988146ffe0d41', 'request_id': 'req-c43878fd-81c8-41fa-b646-d83af6ec8fa5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.546450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a381d1a27fa04724b00988146ffe0d41', 'request_id': 'req-c43878fd-81c8-41fa-b646-d83af6ec8fa5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.558759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] The token cache is being invalidate because project a381d1a27fa04724b00988146ffe0d41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:16.559050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.559810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c43878fd-81c8-41fa-b646-d83af6ec8fa5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:16.561447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1103/2204] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:16 2022] DELETE /identity/v3/projects/a381d1a27fa04724b00988146ffe0d41 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:15:16.566888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b755e-73be-4030-9d5e-9160db45ad9b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.567097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b755e-73be-4030-9d5e-9160db45ad9b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.567401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b755e-73be-4030-9d5e-9160db45ad9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.581421 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4284e4-d686-444f-9d44-54848460d4ea admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.587306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-179b755e-73be-4030-9d5e-9160db45ad9b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:16.588722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-179b755e-73be-4030-9d5e-9160db45ad9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.639672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.640282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.640564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.640836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.641486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.643460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.654765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1104/2205] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:16.663822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f0b9a4-d846-4b9f-838f-36f49e9b2493 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.664106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f0b9a4-d846-4b9f-838f-36f49e9b2493 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.664493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1f0b9a4-d846-4b9f-838f-36f49e9b2493 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.673873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1102/2206] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/users => generated 285 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:16.675087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:16.678163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.679774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.691761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e1f0b9a4-d846-4b9f-838f-36f49e9b2493 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:16.692972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de4cd3ea-7eff-495b-bf3f-756ef4e947e7 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.693798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e1f0b9a4-d846-4b9f-838f-36f49e9b2493 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.697874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.698562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.698836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.699099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.699738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.701385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.702758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1105/2207] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:16.707064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f0b9a4-d846-4b9f-838f-36f49e9b2493 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.711943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1103/2208] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:16 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:16.716368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1f0b9a4-d846-4b9f-838f-36f49e9b2493 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.716735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.721171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.721749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.722025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.722299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.722945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:15:16.723244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.725191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.727985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cf02882-87f6-4caa-9984-a50137a265ed admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.733500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.734140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.734414 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.734683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.735324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.736909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.737655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1106/2209] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:15:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:15:16.741610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.750546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1104/2210] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:16 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:15:16.751476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.752839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3e223d-d0fa-46f6-9bd6-593e7cfdc993 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.758125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.758884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.759156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.759420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.760051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.761034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9851d196-5df5-41a1-8a2a-9241e48d0313 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.762672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.765568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eb25cab-4434-4218-b5e3-d5488df2e038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.766230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb25cab-4434-4218-b5e3-d5488df2e038 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.766522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb25cab-4434-4218-b5e3-d5488df2e038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.766777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eb25cab-4434-4218-b5e3-d5488df2e038 admin admin] PATH_INFO: `/v3/projects/9cb8e79eba8b4680ae6f234bb701c1c1/users/9fb40177898e4e69abdb64dab3724a6c/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.781908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb25cab-4434-4218-b5e3-d5488df2e038 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9cb8e79eba8b4680ae6f234bb701c1c1, user_id=9fb40177898e4e69abdb64dab3724a6c, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.784756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eb25cab-4434-4218-b5e3-d5488df2e038 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.792709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1107/2211] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:16.794292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:16.797628 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.799903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1105/2212] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:16 2022] PUT /identity/v3/projects/9cb8e79eba8b4680ae6f234bb701c1c1/users/9fb40177898e4e69abdb64dab3724a6c/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:16.867419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfa0109a-5663-41de-8ec7-27723bd3ec18 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.871915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.872399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.872680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.901201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] MFA Rules not processed for user `9fb40177898e4e69abdb64dab3724a6c`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:16.902568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.926637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.927241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.927514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.927776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.928449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.930402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.958836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1108/2213] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/users => generated 292 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:16.960505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.969072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1106/2214] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:16 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:16.975490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c69bc41-68b5-4809-a8c5-3e357ea26acd admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:16.980596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:16.981262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:16.981542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:16.981682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:16.981789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:16.982484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:16.983347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:16.984077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:16.994792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1109/2215] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:16 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:16.998905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.009007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69bd6e1d-d1dd-42ea-814a-08729b49cb86 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.013959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.014581 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.014856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.015121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.015764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.017348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.028753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1110/2216] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:16 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:15:17.033019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.043894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b438fe25-5682-4ec0-8153-75aeb94e3aa2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.046215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c249d19-b02d-4a52-86f0-6684335c235e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.049090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.049860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.050250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c62f88ea-e907-4df0-ad96-36f51c70da2a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.050386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.050708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] PATH_INFO: `/v3/projects/d9a14c91967144eebf3f6bd0b8259e23/users/994ea9700d514fe8b034b59ecc26cc4a/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.050918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f88ea-e907-4df0-ad96-36f51c70da2a service glance] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.051197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f88ea-e907-4df0-ad96-36f51c70da2a service glance] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.051601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c62f88ea-e907-4df0-ad96-36f51c70da2a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.066356 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62f88ea-e907-4df0-ad96-36f51c70da2a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.067811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c62f88ea-e907-4df0-ad96-36f51c70da2a service glance] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.072706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=d9a14c91967144eebf3f6bd0b8259e23, user_id=994ea9700d514fe8b034b59ecc26cc4a, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.075558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.078172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1107/2217] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:16 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:17.097040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1111/2218] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:17 2022] PUT /identity/v3/projects/d9a14c91967144eebf3f6bd0b8259e23/users/994ea9700d514fe8b034b59ecc26cc4a/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:15:17.111029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5eec59-a499-4261-b2d1-0ca6c325522a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.111316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5eec59-a499-4261-b2d1-0ca6c325522a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.111591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df5eec59-a499-4261-b2d1-0ca6c325522a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.143448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-df5eec59-a499-4261-b2d1-0ca6c325522a None None] MFA Rules not processed for user `994ea9700d514fe8b034b59ecc26cc4a`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:17.144198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-df5eec59-a499-4261-b2d1-0ca6c325522a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:17.230283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1108/2219] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:17 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:17.235258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.250387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:17.316882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ee93b37-3526-4242-bcbe-d1c463ce14ae admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.326674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.329394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.329826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.330523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.331383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.334356 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.390264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1112/2220] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:17 2022] POST /identity/v3/projects => generated 383 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:17.394212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5eec59-a499-4261-b2d1-0ca6c325522a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.423608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df5eec59-a499-4261-b2d1-0ca6c325522a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.454024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:17.519128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.520092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.520491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.520776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.521614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.524129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.552585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1109/2221] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:17 2022] POST /identity/v3/users => generated 298 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:17.556503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.569182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a47e6266-0dd2-410f-9707-e6f800d67ad6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.575242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.575924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.576510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.576510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.577319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.579087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.591295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1113/2222] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:17 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:15:17.595638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.608643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd14a484-0248-45c3-b7e7-5c44ce37d797 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.613818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.614611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.615104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.615451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] PATH_INFO: `/v3/projects/a191715d3b7342a88a30b68fc09554cb/users/2ae4ca6112ef4a899c976a93e68eb53d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.635540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a191715d3b7342a88a30b68fc09554cb, user_id=2ae4ca6112ef4a899c976a93e68eb53d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.640388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.674034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1110/2223] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:17 2022] PUT /identity/v3/projects/a191715d3b7342a88a30b68fc09554cb/users/2ae4ca6112ef4a899c976a93e68eb53d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 79 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:17.679903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.697446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd0aa3e0-8b0b-4072-abd6-4fa143566b3e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.762950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a6ff15f-7c96-4813-8409-73057064ce26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.763564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6ff15f-7c96-4813-8409-73057064ce26 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.763841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6ff15f-7c96-4813-8409-73057064ce26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.764112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a6ff15f-7c96-4813-8409-73057064ce26 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.764756 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6ff15f-7c96-4813-8409-73057064ce26 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.766397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a6ff15f-7c96-4813-8409-73057064ce26 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.777516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1114/2224] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:17 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:17.784839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.796194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b94857d5-9cc1-46c8-bb67-0505ba6f4226 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:17.802961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:17.803624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.803900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.804171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] PATH_INFO: `/v3/projects/a191715d3b7342a88a30b68fc09554cb/users/2ae4ca6112ef4a899c976a93e68eb53d/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.806125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a191715d3b7342a88a30b68fc09554cb, user_id=2ae4ca6112ef4a899c976a93e68eb53d, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:17.809365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:17.830960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1111/2225] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:17 2022] PUT /identity/v3/projects/a191715d3b7342a88a30b68fc09554cb/users/2ae4ca6112ef4a899c976a93e68eb53d/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:17.842129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f162b86-24bd-4a64-a8cf-1dfd66a42bbf None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:17.842472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f162b86-24bd-4a64-a8cf-1dfd66a42bbf None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:17.842862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f162b86-24bd-4a64-a8cf-1dfd66a42bbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:17.877774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5f162b86-24bd-4a64-a8cf-1dfd66a42bbf None None] MFA Rules not processed for user `2ae4ca6112ef4a899c976a93e68eb53d`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:17.879026 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f162b86-24bd-4a64-a8cf-1dfd66a42bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:17.965814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1115/2226] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:17 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:17.977410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:17.979263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:18.043540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1d43bd4-db75-4a2b-86fd-7a4f1659891f admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:18.048435 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:18.049068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:18.049340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:18.049603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:18.061400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:18.062898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:18.079176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1112/2227] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:17 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:15:18.218288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f162b86-24bd-4a64-a8cf-1dfd66a42bbf None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:18.233657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f162b86-24bd-4a64-a8cf-1dfd66a42bbf None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:18.241101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d40340-991e-46b9-9529-532c3b6a5a34 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:18.242135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d40340-991e-46b9-9529-532c3b6a5a34 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:18.244382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d40340-991e-46b9-9529-532c3b6a5a34 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:18.244812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d40340-991e-46b9-9529-532c3b6a5a34 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:18.250091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:18.331340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d40340-991e-46b9-9529-532c3b6a5a34 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:18.333705 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d40340-991e-46b9-9529-532c3b6a5a34 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:18.349511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1116/2228] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:18 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:22.140805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:22.208038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5060198-1fb7-44f6-b4a9-ad993389f1d2 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:22.212539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:22.213189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:22.213460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:22.213728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] PATH_INFO: `/v3/users/994ea9700d514fe8b034b59ecc26cc4a` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:22.214877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=994ea9700d514fe8b034b59ecc26cc4a)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:22.217104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:22.239122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:22.267775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '994ea9700d514fe8b034b59ecc26cc4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.277629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '994ea9700d514fe8b034b59ecc26cc4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.278492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '994ea9700d514fe8b034b59ecc26cc4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.279532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '994ea9700d514fe8b034b59ecc26cc4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.289215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '994ea9700d514fe8b034b59ecc26cc4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.308998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1113/2229] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:22 2022] DELETE /identity/v3/users/994ea9700d514fe8b034b59ecc26cc4a => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:22.313846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:22.314466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:22.314466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:22.335848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:22.337090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:22.406260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1117/2230] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:22.415916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:22.417442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:22.481023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37c4a957-f216-40a5-8fe8-ab8893aa60b3 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:22.485343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:22.485947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:22.486248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:22.486669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:22.497329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:22.499470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:22.509821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1114/2231] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:22.795244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:22.796733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:22.811821 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65e87c4a-7567-4525-9e56-307c311bf0c0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:22.821403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:22.829240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:22.829527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:22.829818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] PATH_INFO: `/v3/projects/d9a14c91967144eebf3f6bd0b8259e23` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:22.831235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] RBAC: Authorizing `identity:delete_project(project_id=d9a14c91967144eebf3f6bd0b8259e23)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:22.833286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:22.887348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd9a14c91967144eebf3f6bd0b8259e23', 'request_id': 'req-0c1056c9-070f-4561-9f26-701434c3df1b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.897920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd9a14c91967144eebf3f6bd0b8259e23', 'request_id': 'req-0c1056c9-070f-4561-9f26-701434c3df1b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.905253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd9a14c91967144eebf3f6bd0b8259e23', 'request_id': 'req-0c1056c9-070f-4561-9f26-701434c3df1b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.916461 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] The token cache is being invalidate because project d9a14c91967144eebf3f6bd0b8259e23 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:22.916807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.917424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:22.919017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1118/2232] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:22 2022] DELETE /identity/v3/projects/d9a14c91967144eebf3f6bd0b8259e23 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:22.923201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:22.933724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2675ddf8-8634-49b7-8d57-7dd8e3aa1ed0 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:22.996014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:22.996669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:22.996942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:22.997206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] PATH_INFO: `/v3/users/2ae4ca6112ef4a899c976a93e68eb53d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:22.998445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2ae4ca6112ef4a899c976a93e68eb53d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:23.000928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:23.055973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=14,keystone:SELECT=85,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:23.063369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2ae4ca6112ef4a899c976a93e68eb53d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:23.072146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2ae4ca6112ef4a899c976a93e68eb53d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:23.073014 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2ae4ca6112ef4a899c976a93e68eb53d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:23.073644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2ae4ca6112ef4a899c976a93e68eb53d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:23.086042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1c177ca-20a6-4b49-bdaa-a6144cce31c8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2ae4ca6112ef4a899c976a93e68eb53d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:23.103415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1115/2233] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:22 2022] DELETE /identity/v3/users/2ae4ca6112ef4a899c976a93e68eb53d => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:23.831937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:23.833415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:23.847510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1056c9-070f-4561-9f26-701434c3df1b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:23.883890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:23.921697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:23.922742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:23.923199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:23.924263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] PATH_INFO: `/v3/projects/a191715d3b7342a88a30b68fc09554cb` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:23.926159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a191715d3b7342a88a30b68fc09554cb)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:23.928372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:23.988861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a191715d3b7342a88a30b68fc09554cb', 'request_id': 'req-70908b21-89cc-4918-997b-bfb4310f4300'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:23.997678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a191715d3b7342a88a30b68fc09554cb', 'request_id': 'req-70908b21-89cc-4918-997b-bfb4310f4300'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:24.004874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a191715d3b7342a88a30b68fc09554cb', 'request_id': 'req-70908b21-89cc-4918-997b-bfb4310f4300'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:24.015509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] The token cache is being invalidate because project a191715d3b7342a88a30b68fc09554cb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:24.015794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:24.016439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-70908b21-89cc-4918-997b-bfb4310f4300 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:24.017945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1119/2234] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:23 2022] DELETE /identity/v3/projects/a191715d3b7342a88a30b68fc09554cb => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:24.024678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a704a73-f7a2-46dc-a0a9-0bef2dbbe002 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.025193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a704a73-f7a2-46dc-a0a9-0bef2dbbe002 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.025193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a704a73-f7a2-46dc-a0a9-0bef2dbbe002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.047961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-4a704a73-f7a2-46dc-a0a9-0bef2dbbe002 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:24.049386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4a704a73-f7a2-46dc-a0a9-0bef2dbbe002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:24.114844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1116/2235] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:24.123639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a3d45b-0b1e-4864-ba4b-3a78a629a43a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.123992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a3d45b-0b1e-4864-ba4b-3a78a629a43a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.124523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a3d45b-0b1e-4864-ba4b-3a78a629a43a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.149000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-40a3d45b-0b1e-4864-ba4b-3a78a629a43a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:24.150244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40a3d45b-0b1e-4864-ba4b-3a78a629a43a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:24.159732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1120/2236] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:24.164119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a704a73-f7a2-46dc-a0a9-0bef2dbbe002 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.174080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a704a73-f7a2-46dc-a0a9-0bef2dbbe002 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.179275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.179862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.180138 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.180399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.181044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:15:24.181341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.183002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.195895 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1117/2237] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:15:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:15:24.200359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a3d45b-0b1e-4864-ba4b-3a78a629a43a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.210205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a3d45b-0b1e-4864-ba4b-3a78a629a43a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.215079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.215681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.215973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.216234 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.216883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.218855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.259012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1121/2238] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:24 2022] POST /identity/v3/projects => generated 387 bytes in 59 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:24.263223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.273840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ec4e80f-7592-4297-b6b1-b0eefb02f346 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.356068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.356834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.357117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.357389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.358058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.360115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.401684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1118/2239] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:24 2022] POST /identity/v3/users => generated 294 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:24.402914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.423422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3486e76-ca9c-449d-a5bb-7d3d3fcf5017 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.428133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.428762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.429034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.429300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.429952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.431683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.446332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1122/2240] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:24.447648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.458893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c5eeea-3d49-45f1-9e93-7c0e79b878d4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.466833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c1b852f-9a0b-4bae-a007-1f12e06db8e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.467664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b852f-9a0b-4bae-a007-1f12e06db8e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.468080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b852f-9a0b-4bae-a007-1f12e06db8e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.468360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c1b852f-9a0b-4bae-a007-1f12e06db8e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.469049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1b852f-9a0b-4bae-a007-1f12e06db8e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.470902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c1b852f-9a0b-4bae-a007-1f12e06db8e6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.482400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1119/2241] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:24.487035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.500164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed69100d-8718-4e4b-b9d1-a46250108cb9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.504926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.505573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.505844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.506138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] PATH_INFO: `/v3/projects/547a8410eeee465ba00978ab64777f10/users/50c8f43df77741d2b6bfad9a2501a214/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.521578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=547a8410eeee465ba00978ab64777f10, user_id=50c8f43df77741d2b6bfad9a2501a214, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.525418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.546839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1123/2242] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:24 2022] PUT /identity/v3/projects/547a8410eeee465ba00978ab64777f10/users/50c8f43df77741d2b6bfad9a2501a214/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:24.558498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13ee318-07a4-48de-80fa-7160d3031d3c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.558867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13ee318-07a4-48de-80fa-7160d3031d3c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.559208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13ee318-07a4-48de-80fa-7160d3031d3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.590367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c13ee318-07a4-48de-80fa-7160d3031d3c None None] MFA Rules not processed for user `50c8f43df77741d2b6bfad9a2501a214`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:24.591968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c13ee318-07a4-48de-80fa-7160d3031d3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:24.605117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:24.667969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1120/2243] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:24 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:24.673853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.733443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:24.739177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95c24e6-501d-4553-89f4-e2b97d5cbd48 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.745811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.746476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.746755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.747027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.747682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.751075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.784309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1124/2244] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:24 2022] POST /identity/v3/projects => generated 387 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:24.788706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13ee318-07a4-48de-80fa-7160d3031d3c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.802257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c13ee318-07a4-48de-80fa-7160d3031d3c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.864534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.865075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.865373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.865640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.866306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.868253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.892918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1121/2245] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:24 2022] POST /identity/v3/users => generated 300 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:24.897138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.908495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe9ff83-e4e1-4059-9898-8989ed0dba14 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.913771 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.914424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.914699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.914953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.915591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.917247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.928269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1125/2246] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:24.932725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.943269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a611bfb-1504-473e-ad19-6a70ac639b69 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:24.948271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:24.948928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:24.949203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:24.949480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] PATH_INFO: `/v3/projects/a857b59c7c8a404e8abd8e571719e4db/users/c58eb7eddfe945179391b0fd8ebe0828/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:24.966663 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a857b59c7c8a404e8abd8e571719e4db, user_id=c58eb7eddfe945179391b0fd8ebe0828, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:24.969427 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:24.984568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1122/2247] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:24 2022] PUT /identity/v3/projects/a857b59c7c8a404e8abd8e571719e4db/users/c58eb7eddfe945179391b0fd8ebe0828/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:24.988324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:24.998906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27a9c6a-4639-40cd-a636-4f900dc014cc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:25.073036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051004eb-a716-405e-9303-430267186d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:25.073655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051004eb-a716-405e-9303-430267186d0f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:25.073940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051004eb-a716-405e-9303-430267186d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:25.074289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051004eb-a716-405e-9303-430267186d0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:25.075339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051004eb-a716-405e-9303-430267186d0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:25.076541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051004eb-a716-405e-9303-430267186d0f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:25.087887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1126/2248] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:24 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:25.092066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:25.103471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b6680f-7083-4ddc-b468-69865e4e4df4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:25.108833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:25.109665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:25.110047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:25.110406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] PATH_INFO: `/v3/projects/a857b59c7c8a404e8abd8e571719e4db/users/c58eb7eddfe945179391b0fd8ebe0828/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:25.113368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a857b59c7c8a404e8abd8e571719e4db, user_id=c58eb7eddfe945179391b0fd8ebe0828, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:25.116361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:25.129935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1123/2249] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:25 2022] PUT /identity/v3/projects/a857b59c7c8a404e8abd8e571719e4db/users/c58eb7eddfe945179391b0fd8ebe0828/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:25.141368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350bcca8-5572-4cbf-bba0-dd4b874679cd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:25.141636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350bcca8-5572-4cbf-bba0-dd4b874679cd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:25.141907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350bcca8-5572-4cbf-bba0-dd4b874679cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:25.179911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-350bcca8-5572-4cbf-bba0-dd4b874679cd None None] MFA Rules not processed for user `c58eb7eddfe945179391b0fd8ebe0828`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:25.181266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-350bcca8-5572-4cbf-bba0-dd4b874679cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:25.255292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1127/2250] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:25 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:25.266330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:25.268387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:25.332402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e05c806c-116a-4ef1-bcf1-fd1cd4a0c052 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:25.336443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1170d155-4061-4d9d-a9fc-18cdb185bb8b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:25.337183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170d155-4061-4d9d-a9fc-18cdb185bb8b service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:25.337737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170d155-4061-4d9d-a9fc-18cdb185bb8b service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:25.338142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1170d155-4061-4d9d-a9fc-18cdb185bb8b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:25.349566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170d155-4061-4d9d-a9fc-18cdb185bb8b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:25.351083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1170d155-4061-4d9d-a9fc-18cdb185bb8b service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:25.360712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1124/2251] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:25 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:25.510692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350bcca8-5572-4cbf-bba0-dd4b874679cd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:25.563358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:25.575865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350bcca8-5572-4cbf-bba0-dd4b874679cd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:25.581300 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:25.581961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:25.582268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:25.582546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] PATH_INFO: `/v3/users/50c8f43df77741d2b6bfad9a2501a214` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:25.583710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] RBAC: Authorizing `identity:delete_user(user_id=50c8f43df77741d2b6bfad9a2501a214)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:25.585930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:25.651123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50c8f43df77741d2b6bfad9a2501a214'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:25.661161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50c8f43df77741d2b6bfad9a2501a214'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:25.661896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50c8f43df77741d2b6bfad9a2501a214'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:25.662535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50c8f43df77741d2b6bfad9a2501a214'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:25.672942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50c8f43df77741d2b6bfad9a2501a214'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:25.690150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1128/2252] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:25 2022] DELETE /identity/v3/users/50c8f43df77741d2b6bfad9a2501a214 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:25.695591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:25.695968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:25.696326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:25.718860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:25.720066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:25.725216 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:25.803440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1125/2253] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:25.813633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:25.815318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:25.879539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e9354e-242a-467d-8903-a42678d23629 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:25.883840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:25.884449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:25.884722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:25.884989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:25.894553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:25.896036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:25.907053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1129/2254] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:26.201581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:26.203542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:26.242543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95415f00-bf0e-4eae-8bdd-67f51636033e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:26.248033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:26.248962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:26.249266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:26.249645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] PATH_INFO: `/v3/projects/547a8410eeee465ba00978ab64777f10` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:26.251098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] RBAC: Authorizing `identity:delete_project(project_id=547a8410eeee465ba00978ab64777f10)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:26.254340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:26.310177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '547a8410eeee465ba00978ab64777f10', 'request_id': 'req-63430b41-9f4c-429a-897b-ce37e7da880d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.321388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '547a8410eeee465ba00978ab64777f10', 'request_id': 'req-63430b41-9f4c-429a-897b-ce37e7da880d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.328256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '547a8410eeee465ba00978ab64777f10', 'request_id': 'req-63430b41-9f4c-429a-897b-ce37e7da880d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.343413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] The token cache is being invalidate because project 547a8410eeee465ba00978ab64777f10 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:26.343413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.344425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.346252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1126/2255] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:26 2022] DELETE /identity/v3/projects/547a8410eeee465ba00978ab64777f10 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:26.350591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:26.361347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e203cd9c-5881-49e2-83b2-d8e6f4d0a5f5 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:26.432562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:26.433097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:26.433375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:26.433650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] PATH_INFO: `/v3/users/c58eb7eddfe945179391b0fd8ebe0828` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:26.435293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c58eb7eddfe945179391b0fd8ebe0828)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:26.438434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:26.457357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:26.499591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c58eb7eddfe945179391b0fd8ebe0828'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.507768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c58eb7eddfe945179391b0fd8ebe0828'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.508563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c58eb7eddfe945179391b0fd8ebe0828'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.509307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c58eb7eddfe945179391b0fd8ebe0828'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.519312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e218b7c5-530f-4eb6-b76f-21595fe1ee08 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c58eb7eddfe945179391b0fd8ebe0828'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:26.540501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1130/2256] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:26 2022] DELETE /identity/v3/users/c58eb7eddfe945179391b0fd8ebe0828 => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:26.932027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:26.933481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:26.947854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63430b41-9f4c-429a-897b-ce37e7da880d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.006023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.006903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.007041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.007228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] PATH_INFO: `/v3/projects/a857b59c7c8a404e8abd8e571719e4db` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.008651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a857b59c7c8a404e8abd8e571719e4db)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.010432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.051746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:27.063341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a857b59c7c8a404e8abd8e571719e4db', 'request_id': 'req-bef13e4b-b1cb-4b6b-8ab8-086b90917090'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:27.072311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a857b59c7c8a404e8abd8e571719e4db', 'request_id': 'req-bef13e4b-b1cb-4b6b-8ab8-086b90917090'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:27.080057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a857b59c7c8a404e8abd8e571719e4db', 'request_id': 'req-bef13e4b-b1cb-4b6b-8ab8-086b90917090'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:27.091400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] The token cache is being invalidate because project a857b59c7c8a404e8abd8e571719e4db was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:27.091692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:27.092348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bef13e4b-b1cb-4b6b-8ab8-086b90917090 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:27.094659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1127/2257] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:26 2022] DELETE /identity/v3/projects/a857b59c7c8a404e8abd8e571719e4db => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:27.099917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfb3570-ff0a-4ed2-bd6f-6cb85dbcdaac None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.100414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfb3570-ff0a-4ed2-bd6f-6cb85dbcdaac None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.100519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cfb3570-ff0a-4ed2-bd6f-6cb85dbcdaac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.124761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3cfb3570-ff0a-4ed2-bd6f-6cb85dbcdaac None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:27.126228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3cfb3570-ff0a-4ed2-bd6f-6cb85dbcdaac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:27.195936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1131/2258] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:27.204599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6662e51-e522-43ed-b7b5-61ebff69fd1a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.205181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6662e51-e522-43ed-b7b5-61ebff69fd1a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.205181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6662e51-e522-43ed-b7b5-61ebff69fd1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.225638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c6662e51-e522-43ed-b7b5-61ebff69fd1a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:27.227531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c6662e51-e522-43ed-b7b5-61ebff69fd1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:27.240681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1128/2259] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:27.242070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfb3570-ff0a-4ed2-bd6f-6cb85dbcdaac None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.255486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cfb3570-ff0a-4ed2-bd6f-6cb85dbcdaac None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.260652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.261389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.261519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.261788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.262496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:15:27.262795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.264379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.278816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1132/2260] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:15:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:15:27.286208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6662e51-e522-43ed-b7b5-61ebff69fd1a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.292291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6662e51-e522-43ed-b7b5-61ebff69fd1a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.297559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.298206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.298514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.298789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.299432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.301380 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.332452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1129/2261] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:27 2022] POST /identity/v3/projects => generated 379 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:27.337384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.348977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7809d930-b03b-4b81-983e-2ae30f843f3a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.416806 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.417417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.417689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.417956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.418617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.420556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.450267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1133/2262] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:27 2022] POST /identity/v3/users => generated 290 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:27.454502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.465159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-260207bb-0ea1-4f0c-93d1-73b5f92e836c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.470038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.470653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.471024 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.471178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.471818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.473430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.485019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1130/2263] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:15:27.489097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.500825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c44bbba-044f-45ca-8fd0-a924085c827d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.506736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5590bde1-0e69-446f-86bc-024cf8a2b6b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.507346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5590bde1-0e69-446f-86bc-024cf8a2b6b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.507618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5590bde1-0e69-446f-86bc-024cf8a2b6b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.507887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5590bde1-0e69-446f-86bc-024cf8a2b6b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.508493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5590bde1-0e69-446f-86bc-024cf8a2b6b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.510141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5590bde1-0e69-446f-86bc-024cf8a2b6b7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.512832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:27.525527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1134/2264] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:27 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:27.529449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.540850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de13ae1-8bda-44dc-bc75-8371a2f18f21 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.561116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.561116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.561116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.561116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] PATH_INFO: `/v3/projects/b2d42064b4b04b4bbc2286f5ec32d1ee/users/1ed5f90d65ec42fdb9bcb9875f70c72f/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.568718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] RBAC: Authorizing `identity:create_grant(project_id=b2d42064b4b04b4bbc2286f5ec32d1ee, user_id=1ed5f90d65ec42fdb9bcb9875f70c72f, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.571449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.587261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1131/2265] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:27 2022] PUT /identity/v3/projects/b2d42064b4b04b4bbc2286f5ec32d1ee/users/1ed5f90d65ec42fdb9bcb9875f70c72f/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:27.598200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e1b54-c75b-4e06-a02b-8edbde77ae7e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.598502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e1b54-c75b-4e06-a02b-8edbde77ae7e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.598826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f42e1b54-c75b-4e06-a02b-8edbde77ae7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.634589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f42e1b54-c75b-4e06-a02b-8edbde77ae7e None None] MFA Rules not processed for user `1ed5f90d65ec42fdb9bcb9875f70c72f`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:27.635830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f42e1b54-c75b-4e06-a02b-8edbde77ae7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:27.708863 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1135/2266] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:27.713443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.777498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad914d4c-b7ff-41dd-8aca-3036a8601cac admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.782631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.783249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.783531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.783798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.784447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.786444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.828494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1132/2267] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:27 2022] POST /identity/v3/projects => generated 379 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:27.829827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e1b54-c75b-4e06-a02b-8edbde77ae7e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.849644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f42e1b54-c75b-4e06-a02b-8edbde77ae7e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.922480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.923055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.923277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.923600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.924226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.926331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.950996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1136/2268] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:27 2022] POST /identity/v3/users => generated 290 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:27.956778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:27.968452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71695fc-ccd7-4f82-a8bd-608d5c59876a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:27.973086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:27.973682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:27.973955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:27.974667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:27.975310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:27.976976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:27.989317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1133/2269] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:27 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:27.993320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.004161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1f9f87a-370b-43b5-9b96-86772906bf49 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.011088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e3ca67-c6b9-465f-83c4-f2efdd2cbba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.011699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3ca67-c6b9-465f-83c4-f2efdd2cbba6 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.011976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3ca67-c6b9-465f-83c4-f2efdd2cbba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.012242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e3ca67-c6b9-465f-83c4-f2efdd2cbba6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.012936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e3ca67-c6b9-465f-83c4-f2efdd2cbba6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.014568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e3ca67-c6b9-465f-83c4-f2efdd2cbba6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.027059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1137/2270] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:28.031751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.042595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdf312c6-e776-4ebd-8ca2-8cf769c06c92 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.047391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.048050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.048334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.048597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] PATH_INFO: `/v3/projects/a098e59f9b2b4e689acb8bd5fc2fe947/users/0c65f80674854d4fbf48b580ab357c15/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.064857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a098e59f9b2b4e689acb8bd5fc2fe947, user_id=0c65f80674854d4fbf48b580ab357c15, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.067602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.085211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1134/2271] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:28 2022] PUT /identity/v3/projects/a098e59f9b2b4e689acb8bd5fc2fe947/users/0c65f80674854d4fbf48b580ab357c15/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:28.096857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0d5575-065f-4bce-9309-8a6fec4451df None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.097141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0d5575-065f-4bce-9309-8a6fec4451df None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.097417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0d5575-065f-4bce-9309-8a6fec4451df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.126450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5a0d5575-065f-4bce-9309-8a6fec4451df None None] MFA Rules not processed for user `0c65f80674854d4fbf48b580ab357c15`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:28.127664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5a0d5575-065f-4bce-9309-8a6fec4451df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:28.192088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:28.203796 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1138/2272] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:28 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:28.208847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.220180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=5,keystone:SELECT=90,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:28.274962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81bcbbd-a41b-4b48-b6a1-b4a52bb18a71 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.279965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.280609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.280916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.281277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.281819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.283767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.318735 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1135/2273] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:28 2022] POST /identity/v3/projects => generated 377 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:28.323121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0d5575-065f-4bce-9309-8a6fec4451df None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.334782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0d5575-065f-4bce-9309-8a6fec4451df None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.394051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.394659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.394936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.395205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.395812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.397748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.425168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1139/2274] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:28 2022] POST /identity/v3/users => generated 295 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:28.427228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.438498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bd98dc-069d-46f6-84fe-1739d1fe9cef admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.444737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.445371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.445647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.445915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.446605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.448341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.461521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1136/2275] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:28.467191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.483900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5fc4459-358d-44d9-8149-2ed59c1d7d3d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.490418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.491091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.491363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.491635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] PATH_INFO: `/v3/projects/fe95b204db874d389e30ab4b63f5d2fd/users/8eab137ea09c46ce8de6e0d10cc31c67/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.508830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe95b204db874d389e30ab4b63f5d2fd, user_id=8eab137ea09c46ce8de6e0d10cc31c67, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.511511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.530100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1140/2276] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:28 2022] PUT /identity/v3/projects/fe95b204db874d389e30ab4b63f5d2fd/users/8eab137ea09c46ce8de6e0d10cc31c67/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:28.534345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.545384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-453eff43-4ee1-43a8-b15f-11508f5926aa admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.609562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67b7c08-1f47-44fa-809d-d14bcff0b412 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.610312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b7c08-1f47-44fa-809d-d14bcff0b412 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.610591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b7c08-1f47-44fa-809d-d14bcff0b412 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.610866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67b7c08-1f47-44fa-809d-d14bcff0b412 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.611498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67b7c08-1f47-44fa-809d-d14bcff0b412 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.613113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67b7c08-1f47-44fa-809d-d14bcff0b412 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.623554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1137/2277] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:28 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:28.627949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.638318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dd3ee5b-5ee9-4558-9a3d-ea5f36619417 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.643446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.644108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.644377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.644646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] PATH_INFO: `/v3/projects/fe95b204db874d389e30ab4b63f5d2fd/users/8eab137ea09c46ce8de6e0d10cc31c67/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.646666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe95b204db874d389e30ab4b63f5d2fd, user_id=8eab137ea09c46ce8de6e0d10cc31c67, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.649361 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.662191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1141/2278] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:28 2022] PUT /identity/v3/projects/fe95b204db874d389e30ab4b63f5d2fd/users/8eab137ea09c46ce8de6e0d10cc31c67/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:28.672733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeabf40b-5316-4bc0-9ca1-82046ebcd354 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.672992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeabf40b-5316-4bc0-9ca1-82046ebcd354 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.673269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeabf40b-5316-4bc0-9ca1-82046ebcd354 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.702748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-eeabf40b-5316-4bc0-9ca1-82046ebcd354 None None] MFA Rules not processed for user `8eab137ea09c46ce8de6e0d10cc31c67`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:28.704003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-eeabf40b-5316-4bc0-9ca1-82046ebcd354 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:28.780515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1138/2279] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:28 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:28.792071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.793780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:28.867523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0616fc37-d0cd-4e3b-aa53-3295da0c6413 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.871231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f850f71b-1fbf-41f7-8428-357cffa272b0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.871914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850f71b-1fbf-41f7-8428-357cffa272b0 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.872258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850f71b-1fbf-41f7-8428-357cffa272b0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.872536 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f850f71b-1fbf-41f7-8428-357cffa272b0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:28.882160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f850f71b-1fbf-41f7-8428-357cffa272b0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:28.883617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f850f71b-1fbf-41f7-8428-357cffa272b0 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:28.893235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1142/2280] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:28 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:28.938964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeabf40b-5316-4bc0-9ca1-82046ebcd354 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:28.947588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:28.952838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeabf40b-5316-4bc0-9ca1-82046ebcd354 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:28.957551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:28.958218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:28.958747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:28.959222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:29.030659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:29.032163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:29.044514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1139/2281] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:28 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:32.526692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f5232-fbeb-45a4-9076-b9b90953a864 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:32.527200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f5232-fbeb-45a4-9076-b9b90953a864 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:32.527200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81f5232-fbeb-45a4-9076-b9b90953a864 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:32.547853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b81f5232-fbeb-45a4-9076-b9b90953a864 None None] MFA Rules not processed for user `0df60784a8b3468ca19ec2cee1d929e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:32.549089 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b81f5232-fbeb-45a4-9076-b9b90953a864 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:32.600611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:32.616201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1143/2282] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:32 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:15:34.175353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:34.238739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21d6634-6ba2-40d0-912f-a58418f52b25 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:34.243791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:34.244311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:34.244542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:34.244810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] PATH_INFO: `/v3/users/1ed5f90d65ec42fdb9bcb9875f70c72f` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:34.246056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ed5f90d65ec42fdb9bcb9875f70c72f)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:34.248128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:34.302639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ed5f90d65ec42fdb9bcb9875f70c72f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.313571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ed5f90d65ec42fdb9bcb9875f70c72f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.314373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ed5f90d65ec42fdb9bcb9875f70c72f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.315250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ed5f90d65ec42fdb9bcb9875f70c72f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.326892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ed5f90d65ec42fdb9bcb9875f70c72f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.349628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1140/2283] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:34 2022] DELETE /identity/v3/users/1ed5f90d65ec42fdb9bcb9875f70c72f => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:34.355109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:34.355571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:34.355836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:34.376279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:34.377937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:34.439155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1144/2284] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:34.448959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:34.450318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:34.477728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:34.509367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ed917ef-e9af-4282-8bed-2003ab732ebd admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:34.513200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:34.513805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:34.514077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:34.514373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:34.523715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:34.525145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:34.534702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1141/2285] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:34.824818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:34.826435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:34.839826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed63f865-6647-45a3-be93-cc5ef4b4a30c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:34.846415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:34.847155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:34.847417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:34.847935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] PATH_INFO: `/v3/projects/b2d42064b4b04b4bbc2286f5ec32d1ee` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:34.850142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b2d42064b4b04b4bbc2286f5ec32d1ee)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:34.852478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:34.911828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b2d42064b4b04b4bbc2286f5ec32d1ee', 'request_id': 'req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.922330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b2d42064b4b04b4bbc2286f5ec32d1ee', 'request_id': 'req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.932111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b2d42064b4b04b4bbc2286f5ec32d1ee', 'request_id': 'req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.945380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] The token cache is being invalidate because project b2d42064b4b04b4bbc2286f5ec32d1ee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:34.945834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.946662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:34.951581 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1145/2286] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:34 2022] DELETE /identity/v3/projects/b2d42064b4b04b4bbc2286f5ec32d1ee => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:34.952975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:34.963796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c798c7fd-642e-4ed1-9146-5da01f2373c2 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:35.021949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:35.022688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:35.022964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:35.023235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] PATH_INFO: `/v3/users/0c65f80674854d4fbf48b580ab357c15` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:35.024381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] RBAC: Authorizing `identity:delete_user(user_id=0c65f80674854d4fbf48b580ab357c15)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:35.026361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:35.079848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0c65f80674854d4fbf48b580ab357c15'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.087458 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0c65f80674854d4fbf48b580ab357c15'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.088031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0c65f80674854d4fbf48b580ab357c15'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.088824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0c65f80674854d4fbf48b580ab357c15'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.102873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0c65f80674854d4fbf48b580ab357c15'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.121532 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1142/2287] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:34 2022] DELETE /identity/v3/users/0c65f80674854d4fbf48b580ab357c15 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:35.462240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:35.465414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:35.496722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47ff0a0e-33f1-4b6f-9a0f-bf61966eaa46 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:35.578537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:35.579557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:35.579845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:35.580123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] PATH_INFO: `/v3/projects/a098e59f9b2b4e689acb8bd5fc2fe947` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:35.581315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a098e59f9b2b4e689acb8bd5fc2fe947)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:35.583299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:35.607427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:35.634838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a098e59f9b2b4e689acb8bd5fc2fe947', 'request_id': 'req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.643598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a098e59f9b2b4e689acb8bd5fc2fe947', 'request_id': 'req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.650719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a098e59f9b2b4e689acb8bd5fc2fe947', 'request_id': 'req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.661370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] The token cache is being invalidate because project a098e59f9b2b4e689acb8bd5fc2fe947 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:35.661656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.662334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.663841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1146/2288] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:35 2022] DELETE /identity/v3/projects/a098e59f9b2b4e689acb8bd5fc2fe947 => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:35.667909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:35.669345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:35.682048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3742c1d6-6d7d-4510-8761-777154d6684b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:35.728102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:35.747373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:35.748273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:35.748504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:35.750362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] PATH_INFO: `/v3/users/8eab137ea09c46ce8de6e0d10cc31c67` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:35.750362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8eab137ea09c46ce8de6e0d10cc31c67)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:35.771466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:35.830499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8eab137ea09c46ce8de6e0d10cc31c67'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.840760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8eab137ea09c46ce8de6e0d10cc31c67'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.840988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8eab137ea09c46ce8de6e0d10cc31c67'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.841470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8eab137ea09c46ce8de6e0d10cc31c67'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.853091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c535e046-c7a2-4a5e-aaa9-1d36df4caca1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8eab137ea09c46ce8de6e0d10cc31c67'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:35.875891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1143/2289] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:15:35 2022] DELETE /identity/v3/users/8eab137ea09c46ce8de6e0d10cc31c67 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:36.173754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.175268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:36.188971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cd02ac-14eb-4cd4-aee3-3e1c1b8998a8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:36.256553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:36.257521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.257892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.258211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] PATH_INFO: `/v3/projects/fe95b204db874d389e30ab4b63f5d2fd` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.259458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe95b204db874d389e30ab4b63f5d2fd)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:36.261584 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:36.315729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe95b204db874d389e30ab4b63f5d2fd', 'request_id': 'req-0f169232-6a38-45b5-aa65-0dd293b98aff'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:36.325460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe95b204db874d389e30ab4b63f5d2fd', 'request_id': 'req-0f169232-6a38-45b5-aa65-0dd293b98aff'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:36.334315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe95b204db874d389e30ab4b63f5d2fd', 'request_id': 'req-0f169232-6a38-45b5-aa65-0dd293b98aff'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:36.345855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] The token cache is being invalidate because project fe95b204db874d389e30ab4b63f5d2fd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:15:36.346316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:36.347027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-0f169232-6a38-45b5-aa65-0dd293b98aff admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:15:36.348551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1147/2290] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:15:36 2022] DELETE /identity/v3/projects/fe95b204db874d389e30ab4b63f5d2fd => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:36.356389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030aafcd-02aa-43b6-b41c-67c0d9db8df1 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.356640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030aafcd-02aa-43b6-b41c-67c0d9db8df1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.356905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-030aafcd-02aa-43b6-b41c-67c0d9db8df1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.377975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-030aafcd-02aa-43b6-b41c-67c0d9db8df1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:36.379214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-030aafcd-02aa-43b6-b41c-67c0d9db8df1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:36.439390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1144/2291] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:36.447949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b4e4b-4a37-4702-9b29-9903173be30e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.448219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b4e4b-4a37-4702-9b29-9903173be30e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.448494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180b4e4b-4a37-4702-9b29-9903173be30e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.468393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-180b4e4b-4a37-4702-9b29-9903173be30e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:36.469873 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-180b4e4b-4a37-4702-9b29-9903173be30e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:36.478564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1148/2292] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:36.482850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030aafcd-02aa-43b6-b41c-67c0d9db8df1 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.492752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-030aafcd-02aa-43b6-b41c-67c0d9db8df1 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:36.497264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:36.497862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.498217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.498480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.499119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:15:36.499412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:36.501162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:36.512798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1145/2293] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:15:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:15:36.517398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180b4e4b-4a37-4702-9b29-9903173be30e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.526661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-180b4e4b-4a37-4702-9b29-9903173be30e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:36.531753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:36.532363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.532699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.533124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.533966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:36.536331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:36.574237 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1149/2294] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:15:36 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:15:36.581530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.608263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3418ff4-b0a3-41a5-be40-c293c1a92995 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:36.686036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:36.698170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:36.698851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.699107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.699357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.700293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:36.702918 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:36.727844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1146/2295] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:15:36 2022] POST /identity/v3/users => generated 291 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:15:36.731930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.742919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8964b5-f2bf-4bbe-8a0e-28c2b0ec85e2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:36.747763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:36.748366 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.748638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.748905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.749550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:36.751199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:36.757147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:36.764245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1150/2296] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:36.766836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.777211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a61f4c-786a-4f7d-8ca6-1e66a6cf7134 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:36.782225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc03d7f-1e44-4bb9-b417-670864c6c889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:36.782779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc03d7f-1e44-4bb9-b417-670864c6c889 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.783049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc03d7f-1e44-4bb9-b417-670864c6c889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.783361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc03d7f-1e44-4bb9-b417-670864c6c889 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.784261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc03d7f-1e44-4bb9-b417-670864c6c889 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:36.785908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc03d7f-1e44-4bb9-b417-670864c6c889 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:36.801113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1147/2297] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:15:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:15:36.802444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.812757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9b6b2e-c375-4f08-a37f-8235f8522894 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:36.817790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:36.818616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.818932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.819228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] PATH_INFO: `/v3/projects/03571a9d7e1e43f1b5eef66eeae55844/users/ffb896a1dae34a9dad20c23433855115/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.841102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] RBAC: Authorizing `identity:create_grant(project_id=03571a9d7e1e43f1b5eef66eeae55844, user_id=ffb896a1dae34a9dad20c23433855115, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:36.844002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:36.867694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1151/2298] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:15:36 2022] PUT /identity/v3/projects/03571a9d7e1e43f1b5eef66eeae55844/users/ffb896a1dae34a9dad20c23433855115/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:15:36.872787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bcc33e-8fc5-4a85-8150-92f9713eef52 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:36.873066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bcc33e-8fc5-4a85-8150-92f9713eef52 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:36.873329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76bcc33e-8fc5-4a85-8150-92f9713eef52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:36.892792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-76bcc33e-8fc5-4a85-8150-92f9713eef52 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:36.894057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-76bcc33e-8fc5-4a85-8150-92f9713eef52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:36.951788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1148/2299] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:36.963011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:36.964744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:37.023582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e20e85-cb7b-4e48-837d-81b9ed482492 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:37.027470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:37.028078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:37.028371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:37.028633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:37.037950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:37.039405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:37.049063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1152/2300] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:37.734472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2649c482-ce6d-4d1b-9a81-3f01dcf1f39c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:37.735020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2649c482-ce6d-4d1b-9a81-3f01dcf1f39c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:37.735115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2649c482-ce6d-4d1b-9a81-3f01dcf1f39c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:37.763724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2649c482-ce6d-4d1b-9a81-3f01dcf1f39c None None] MFA Rules not processed for user `ffb896a1dae34a9dad20c23433855115`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:37.764964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2649c482-ce6d-4d1b-9a81-3f01dcf1f39c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:37.833420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1149/2301] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:37 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:37.845405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:37.846898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:37.928815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5e12c5-4bb8-4638-bb04-472884fcdd4d service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:37.932375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:37.932989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:37.933255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:37.933516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:37.942382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:37.943810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:37.952877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1153/2302] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:37 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:40.384724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:40.385295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:40.385295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:40.405721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] MFA Rules not processed for user `ffb896a1dae34a9dad20c23433855115`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:15:40.406971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:40.414628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1150/2303] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:15:40 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:15:45.118716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:15:45.128146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfad59b-057c-4e92-98f5-7e42c97673be service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:15:45.132395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c401e02a-81e6-431c-9a97-261b22d7a3b4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:15:45.133069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401e02a-81e6-431c-9a97-261b22d7a3b4 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:15:45.133407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401e02a-81e6-431c-9a97-261b22d7a3b4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:15:45.133793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c401e02a-81e6-431c-9a97-261b22d7a3b4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:15:45.135720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c401e02a-81e6-431c-9a97-261b22d7a3b4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:15:45.207322 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c401e02a-81e6-431c-9a97-261b22d7a3b4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:15:45.209405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c401e02a-81e6-431c-9a97-261b22d7a3b4 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:15:45.224297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1154/2304] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:15:45 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:15:50.400182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=73,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:15:55.217100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:11.246441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:11.248025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:11.263001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3242fa61-a8e0-4b68-b36d-f11af041c489 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:11.268302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:11.268928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:11.269198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:11.269469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] PATH_INFO: `/v3/users/9fb40177898e4e69abdb64dab3724a6c` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:11.270624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9fb40177898e4e69abdb64dab3724a6c)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:11.272525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:11.325846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9fb40177898e4e69abdb64dab3724a6c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.333313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9fb40177898e4e69abdb64dab3724a6c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.334072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9fb40177898e4e69abdb64dab3724a6c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.334727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9fb40177898e4e69abdb64dab3724a6c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.344533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9fb40177898e4e69abdb64dab3724a6c'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.363452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1151/2305] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:16:11 2022] DELETE /identity/v3/users/9fb40177898e4e69abdb64dab3724a6c => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:16:11.369114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:11.369672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:11.369672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:11.389371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:11.390590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:11.452984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1155/2306] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:16:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:11.463550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:11.464988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:11.528399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75ac5373-c4ec-4fd4-b42d-9af1896f91f4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:11.532924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a4e06ea-2aeb-4435-88d6-0dfe84149cd7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:11.533533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e06ea-2aeb-4435-88d6-0dfe84149cd7 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:11.533803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e06ea-2aeb-4435-88d6-0dfe84149cd7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:11.534087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a4e06ea-2aeb-4435-88d6-0dfe84149cd7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:11.546071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e06ea-2aeb-4435-88d6-0dfe84149cd7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:11.547517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a4e06ea-2aeb-4435-88d6-0dfe84149cd7 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:11.558479 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1152/2307] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:11.855153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:11.856698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:11.869422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1f004b6-9b0e-4cbb-bb24-71e97eeac31f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:11.878191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:11.878920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:11.879198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:11.879439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] PATH_INFO: `/v3/projects/9cb8e79eba8b4680ae6f234bb701c1c1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:11.880922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9cb8e79eba8b4680ae6f234bb701c1c1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:11.882942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:11.952993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9cb8e79eba8b4680ae6f234bb701c1c1', 'request_id': 'req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.962551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9cb8e79eba8b4680ae6f234bb701c1c1', 'request_id': 'req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.972736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9cb8e79eba8b4680ae6f234bb701c1c1', 'request_id': 'req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.985696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] The token cache is being invalidate because project 9cb8e79eba8b4680ae6f234bb701c1c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:16:11.985967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.986727 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82b73d4e-b092-4a1d-a7ee-3e48c09d0ef6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:11.988262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1156/2308] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:16:11 2022] DELETE /identity/v3/projects/9cb8e79eba8b4680ae6f234bb701c1c1 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:16:11.993844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98112424-ed8a-4767-83d7-c56d69656e3a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:11.994153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98112424-ed8a-4767-83d7-c56d69656e3a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:11.994420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98112424-ed8a-4767-83d7-c56d69656e3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.017644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-98112424-ed8a-4767-83d7-c56d69656e3a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:12.018907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-98112424-ed8a-4767-83d7-c56d69656e3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:12.081631 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1153/2309] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:16:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:12.090041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128daa4-a5ec-4d53-8f9f-fd1947a8c292 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.090315 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128daa4-a5ec-4d53-8f9f-fd1947a8c292 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.090579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2128daa4-a5ec-4d53-8f9f-fd1947a8c292 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.111904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-2128daa4-a5ec-4d53-8f9f-fd1947a8c292 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:12.113023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2128daa4-a5ec-4d53-8f9f-fd1947a8c292 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:12.121557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1157/2310] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:16:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:12.126243 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98112424-ed8a-4767-83d7-c56d69656e3a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:12.137187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98112424-ed8a-4767-83d7-c56d69656e3a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:12.143520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:12.144165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.144437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.144703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.145344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:16:12.145646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:12.147320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:12.161188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1154/2311] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:16:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:16:12.167828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128daa4-a5ec-4d53-8f9f-fd1947a8c292 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:12.179000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2128daa4-a5ec-4d53-8f9f-fd1947a8c292 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:12.184518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:12.185127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.185400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.185665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.186645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:12.188593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:12.225639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1158/2312] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:16:12 2022] POST /identity/v3/projects => generated 391 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:16:12.230634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:12.236050 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:12.242312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abbf9c-732f-4100-8464-0aea2dde6369 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:12.302140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:12.302810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.303033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.303300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.304190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:12.306607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:12.330586 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1155/2313] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:16:12 2022] POST /identity/v3/users => generated 296 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:16:12.334860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:12.345933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d70bec25-dff2-4ee5-97a0-111f212cb4ab admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:12.350910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:12.351529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.351805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.352117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.353101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:12.355018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:12.369247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1159/2314] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:16:12 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:16:12.374692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:12.386537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ddbfc9f-d741-46e9-815b-06e1aad909d0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:12.392815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70e08ab3-1c27-41b4-9f9f-76b4dd9ff5f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:12.393609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e08ab3-1c27-41b4-9f9f-76b4dd9ff5f3 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.393953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e08ab3-1c27-41b4-9f9f-76b4dd9ff5f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.394811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70e08ab3-1c27-41b4-9f9f-76b4dd9ff5f3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.395796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e08ab3-1c27-41b4-9f9f-76b4dd9ff5f3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:12.400041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70e08ab3-1c27-41b4-9f9f-76b4dd9ff5f3 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:12.411121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1156/2315] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:16:12 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:16:12.415427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:12.427359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde527e9-3db9-4355-9b68-10856a05525f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:12.432100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:12.432778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.433051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.433320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] PATH_INFO: `/v3/projects/3644808c0131491e89cd1495177c47de/users/afe7e358a32e453fa0ec9255a9043f9a/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.449019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3644808c0131491e89cd1495177c47de, user_id=afe7e358a32e453fa0ec9255a9043f9a, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:12.451772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:12.469986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1160/2316] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:16:12 2022] PUT /identity/v3/projects/3644808c0131491e89cd1495177c47de/users/afe7e358a32e453fa0ec9255a9043f9a/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:16:12.482743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.484101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.484547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.515953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] MFA Rules not processed for user `afe7e358a32e453fa0ec9255a9043f9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:12.517153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:12.596553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1157/2317] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:16:12 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:12.609230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:12.611358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:12.638543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:12.680797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf4fc2cf-a7b1-48f0-9f4b-43a48f4b80f5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:12.684563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:12.685166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:12.685436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:12.685702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:12.694568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:12.696344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:12.707743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1161/2318] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:12 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:22.592519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=65,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:22.701470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=17 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:44.160556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:44.162034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:44.240228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-891b25fc-c304-4a69-8f67-5c3f3a166f0b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:44.245509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad78232-e659-46ca-8e59-afe97da080ce service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:44.246143 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad78232-e659-46ca-8e59-afe97da080ce service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:44.246415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad78232-e659-46ca-8e59-afe97da080ce service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:44.246679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad78232-e659-46ca-8e59-afe97da080ce service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:44.248472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cad78232-e659-46ca-8e59-afe97da080ce service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:44.259606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad78232-e659-46ca-8e59-afe97da080ce service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:44.261204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad78232-e659-46ca-8e59-afe97da080ce service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:44.271603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1158/2319] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:46.939201 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:47.016429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35a53893-7024-4d47-8d33-a3a9a0840050 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:47.021019 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:47.021640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:47.021911 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:47.022329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] PATH_INFO: `/v3/users/afe7e358a32e453fa0ec9255a9043f9a` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:47.023455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] RBAC: Authorizing `identity:delete_user(user_id=afe7e358a32e453fa0ec9255a9043f9a)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:47.025338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:47.075884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'afe7e358a32e453fa0ec9255a9043f9a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.084103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'afe7e358a32e453fa0ec9255a9043f9a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.084814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'afe7e358a32e453fa0ec9255a9043f9a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.085844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'afe7e358a32e453fa0ec9255a9043f9a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.095309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'afe7e358a32e453fa0ec9255a9043f9a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.116887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1162/2320] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:16:46 2022] DELETE /identity/v3/users/afe7e358a32e453fa0ec9255a9043f9a => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:16:47.121560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:47.122439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:47.122837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:47.148547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:16:47.149777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:47.217223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1159/2321] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:16:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:16:47.227864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:47.229496 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:47.296381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26245d9e-410a-494b-a9d9-be191a82d503 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:47.300247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-688b5e9f-2412-49a1-be88-4df6aa7c42de service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:47.300853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688b5e9f-2412-49a1-be88-4df6aa7c42de service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:47.301123 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688b5e9f-2412-49a1-be88-4df6aa7c42de service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:47.301390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-688b5e9f-2412-49a1-be88-4df6aa7c42de service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:47.314770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688b5e9f-2412-49a1-be88-4df6aa7c42de service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:47.316101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-688b5e9f-2412-49a1-be88-4df6aa7c42de service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:47.326962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1163/2322] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:16:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:16:47.630485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:16:47.632734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:16:47.645375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8878cde-fb6d-471d-bf30-c708bc6248ee None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:16:47.650831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:16:47.651466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:16:47.651738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:16:47.652134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] PATH_INFO: `/v3/projects/3644808c0131491e89cd1495177c47de` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:16:47.653441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3644808c0131491e89cd1495177c47de)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:16:47.656438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:16:47.705987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3644808c0131491e89cd1495177c47de', 'request_id': 'req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.715570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3644808c0131491e89cd1495177c47de', 'request_id': 'req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.722519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3644808c0131491e89cd1495177c47de', 'request_id': 'req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.733470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] The token cache is being invalidate because project 3644808c0131491e89cd1495177c47de was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:16:47.733680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.734338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8f8cc83f-9eda-4359-84f6-dd119ba82cf2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:16:47.735818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1160/2323] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:16:47 2022] DELETE /identity/v3/projects/3644808c0131491e89cd1495177c47de => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:16:57.322153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=79,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:16:57.730106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=86,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:22.492643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:22.493150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:22.493249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:22.513092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] MFA Rules not processed for user `a6111a16f95d4bc58f84d9f60f703d80`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:22.514376 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:22.571417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1164/2324] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:17:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:32.564171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=25 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:42.195106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e0820-5a54-41d8-92d2-4b137e26e2c7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:42.195897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e0820-5a54-41d8-92d2-4b137e26e2c7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:42.195897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d6e0820-5a54-41d8-92d2-4b137e26e2c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:42.220331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5d6e0820-5a54-41d8-92d2-4b137e26e2c7 None None] MFA Rules not processed for user `ffb896a1dae34a9dad20c23433855115`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:42.221505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5d6e0820-5a54-41d8-92d2-4b137e26e2c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:42.280272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1161/2325] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:17:42 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:17:43.140362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:43.141839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:43.218712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-491a7ce9-3d6f-46ce-a4e4-a016c98e125f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:43.225495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:43.226181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:43.226507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:43.226836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:43.228808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:43.241916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:43.243598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:43.253026 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1165/2326] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:43 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:44.512393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67763a77-4c56-4f5e-846d-3fe65d27ef82 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.512937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67763a77-4c56-4f5e-846d-3fe65d27ef82 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.512937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67763a77-4c56-4f5e-846d-3fe65d27ef82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.538267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-67763a77-4c56-4f5e-846d-3fe65d27ef82 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:17:44.539503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-67763a77-4c56-4f5e-846d-3fe65d27ef82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:44.605124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1162/2327] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:17:44 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:17:44.658856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:44.661549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:44.737438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e012d858-e9f1-4a84-a392-219da4b0835d service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:44.742795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:44.743403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:44.743674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:44.743942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:44.752126 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:44.753813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:44.761857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1166/2328] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:17:44 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:17:45.990760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67763a77-4c56-4f5e-846d-3fe65d27ef82 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:46.004592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67763a77-4c56-4f5e-846d-3fe65d27ef82 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:46.010251 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:46.011101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:46.011676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:46.011958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:46.013820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:46.096350 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:46.100035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:46.114252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1163/2329] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:45 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 124 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:46.969846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:46.972012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:47.114369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f67fc83-1386-4540-b1db-7b0103d6608a service swift] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:47.120374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:47.121608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:47.122138 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:47.122632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:47.127441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:47.163082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:47.166824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:47.187848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1167/2330] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:47.334313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:47.335823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:47.353418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae75c627-e5ec-426e-ad18-dd54fd355561 service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:47.357798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:47.359530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:47.360073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:47.360776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:47.363777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:47.381216 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:47.383126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:47.412750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1164/2331] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 75 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:17:54.248015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:17:54.271494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9f59352-8d23-4c97-bdb6-de8dd09e89e0 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:17:54.280782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6bdb5b-6be2-4204-b4cc-21274acbc4a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:17:54.281511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6bdb5b-6be2-4204-b4cc-21274acbc4a8 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:17:54.281852 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6bdb5b-6be2-4204-b4cc-21274acbc4a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:17:54.282235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6bdb5b-6be2-4204-b4cc-21274acbc4a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:17:54.284079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-aa6bdb5b-6be2-4204-b4cc-21274acbc4a8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:17:54.304303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:17:54.372547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6bdb5b-6be2-4204-b4cc-21274acbc4a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:17:54.374399 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6bdb5b-6be2-4204-b4cc-21274acbc4a8 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:17:54.386685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1168/2332] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:17:54 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:17:57.389887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=84 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:03.459725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:03.468569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d89d18-15a4-4b1a-b25d-b64dca98ff1c service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:03.473299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122d5754-364d-45bf-97fe-7b5990da742e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:03.473983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d5754-364d-45bf-97fe-7b5990da742e service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:03.474269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d5754-364d-45bf-97fe-7b5990da742e service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:03.474606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122d5754-364d-45bf-97fe-7b5990da742e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:03.476586 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-122d5754-364d-45bf-97fe-7b5990da742e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:03.487751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d5754-364d-45bf-97fe-7b5990da742e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:03.489191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122d5754-364d-45bf-97fe-7b5990da742e service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:03.497688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1165/2333] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:03.636969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716d28c9-26a7-48c7-a2a8-7fd4a6da1ae2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:03.637494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716d28c9-26a7-48c7-a2a8-7fd4a6da1ae2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:03.637494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-716d28c9-26a7-48c7-a2a8-7fd4a6da1ae2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:03.638854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1169/2334] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:18:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:18:03.657698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28588a88-bc47-4f47-9819-d88c944d5916 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:03.657969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28588a88-bc47-4f47-9819-d88c944d5916 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:03.658246 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28588a88-bc47-4f47-9819-d88c944d5916 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:03.685792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-28588a88-bc47-4f47-9819-d88c944d5916 None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:03.687295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-28588a88-bc47-4f47-9819-d88c944d5916 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:03.706163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1166/2335] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:18:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:03.715931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716d28c9-26a7-48c7-a2a8-7fd4a6da1ae2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:03.717872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-716d28c9-26a7-48c7-a2a8-7fd4a6da1ae2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:03.810431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-716d28c9-26a7-48c7-a2a8-7fd4a6da1ae2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:03.814557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c908efdb-9ece-487d-85b6-877a89eded63 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:03.815161 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c908efdb-9ece-487d-85b6-877a89eded63 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:03.815436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c908efdb-9ece-487d-85b6-877a89eded63 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:03.815706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c908efdb-9ece-487d-85b6-877a89eded63 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:03.827690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c908efdb-9ece-487d-85b6-877a89eded63 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:03.829133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c908efdb-9ece-487d-85b6-877a89eded63 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:03.841578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1170/2336] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:03 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:03.920719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f201239-5df1-4cb7-aee1-38b253b740af None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:03.921078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f201239-5df1-4cb7-aee1-38b253b740af None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:03.921420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f201239-5df1-4cb7-aee1-38b253b740af None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:03.922890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1167/2337] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:18:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:18:03.927752 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbfb788-c823-4d70-b6fe-db23967f1ab0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:03.928221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbfb788-c823-4d70-b6fe-db23967f1ab0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:03.928682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbfb788-c823-4d70-b6fe-db23967f1ab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:03.950414 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-abbfb788-c823-4d70-b6fe-db23967f1ab0 None None] MFA Rules not processed for user `cc2f041fe7194a3da199517b4cdbd265`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:03.951704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-abbfb788-c823-4d70-b6fe-db23967f1ab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:03.958589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1171/2338] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:18:03 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:03.970899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f201239-5df1-4cb7-aee1-38b253b740af None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:03.982715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f201239-5df1-4cb7-aee1-38b253b740af None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:03.986587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32629458-d3ff-4394-845f-691f8c423c0c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:03.987184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32629458-d3ff-4394-845f-691f8c423c0c service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:03.987457 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32629458-d3ff-4394-845f-691f8c423c0c service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:03.987722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32629458-d3ff-4394-845f-691f8c423c0c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:03.995560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32629458-d3ff-4394-845f-691f8c423c0c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:03.997049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32629458-d3ff-4394-845f-691f8c423c0c service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:04.005002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1168/2339] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:03 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 35 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:04.098358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778198e-cbc5-4fc4-9d28-6033a0d3766d None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:04.098645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778198e-cbc5-4fc4-9d28-6033a0d3766d None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:04.098941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e778198e-cbc5-4fc4-9d28-6033a0d3766d None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:04.100327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1172/2340] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:18:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:18:04.106496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:04.106787 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:04.107080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:04.129985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:04.131328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:04.141171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1169/2341] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:18:04 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:18:04.152079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778198e-cbc5-4fc4-9d28-6033a0d3766d None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:04.164477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e778198e-cbc5-4fc4-9d28-6033a0d3766d None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:04.169106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:04.169696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:04.169950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:04.170248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:04.181877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:04.183333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:04.194270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1173/2342] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:04.260007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:04.261686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:04.340453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6540ddb1-e99c-402c-880a-687b8013364d None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:04.346098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7eab01-e868-4cd6-8a44-5ed718a200fe service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:04.346723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7eab01-e868-4cd6-8a44-5ed718a200fe service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:04.347003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7eab01-e868-4cd6-8a44-5ed718a200fe service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:04.347264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7eab01-e868-4cd6-8a44-5ed718a200fe service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:04.349074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3b7eab01-e868-4cd6-8a44-5ed718a200fe service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:04.362656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7eab01-e868-4cd6-8a44-5ed718a200fe service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:04.364448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7eab01-e868-4cd6-8a44-5ed718a200fe service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:04.375860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1170/2343] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:04 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:14.189761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=55 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:14.371125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=50 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:18.734792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:18.746640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e3cb26-c182-46f0-87a4-b55dd461ec4d service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:18.752222 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:18.752981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:18.753318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:18.753647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:18.756298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:18.768007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:18.769434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:18.777747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1174/2344] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:18 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:19.977169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:19.977692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:19.977692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:19.998569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:18:19.999689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:20.008107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1171/2345] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:18:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:18:20.024941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:20.043900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49a90914-8e11-4145-931a-e0a38041316f service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:20.049717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:20.050369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:20.050636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:20.050900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:20.059532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:20.061099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:20.069245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1175/2346] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:18:20 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:18:29.993004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:30.066063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=12 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:18:50.732438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:50.734039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:50.751403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece20e33-ee0f-482d-95e5-23b9b0f2e158 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:50.757987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7398fedf-1e73-45f0-a26d-131de0ed264e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:50.759047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398fedf-1e73-45f0-a26d-131de0ed264e service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:50.759359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398fedf-1e73-45f0-a26d-131de0ed264e service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:50.759607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7398fedf-1e73-45f0-a26d-131de0ed264e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:50.762260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7398fedf-1e73-45f0-a26d-131de0ed264e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:50.776568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7398fedf-1e73-45f0-a26d-131de0ed264e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:50.778229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7398fedf-1e73-45f0-a26d-131de0ed264e service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:50.789373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1172/2347] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:50 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:18:59.752058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:18:59.764558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba87eb4c-c4e4-4951-bfcd-3b743d9f485b service swift] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:18:59.769857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:18:59.771163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:18:59.771479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:18:59.771747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:18:59.773454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:18:59.785626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:18:59.787263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:18:59.798060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1176/2348] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:18:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:00.784276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:02.978109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a459e81-afd3-4f29-8c49-735905ad7781 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:02.978651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a459e81-afd3-4f29-8c49-735905ad7781 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:02.978651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a459e81-afd3-4f29-8c49-735905ad7781 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:03.003719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8a459e81-afd3-4f29-8c49-735905ad7781 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:03.004857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8a459e81-afd3-4f29-8c49-735905ad7781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:03.012501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1173/2349] 10.222.0.61 () {58 vars in 967 bytes} [Thu Sep 29 02:19:02 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:19:03.037105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:03.048933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7c020f-dea0-4d53-b8f2-a1aedace74bb service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:03.079969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:03.080960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:03.080960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:03.081111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:03.095080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:03.096687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:03.106619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1177/2350] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:19:03 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 70 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:19:06.298460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a459e81-afd3-4f29-8c49-735905ad7781 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:06.313874 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a459e81-afd3-4f29-8c49-735905ad7781 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:06.321067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:06.321677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:06.321950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:06.322245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:06.324287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:06.383924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:06.385529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:06.395128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1174/2351] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:19:06 2022] GET /identity/v3/auth/tokens => generated 3270 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:13.102787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=12 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:16.391140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=31 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:21.278337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:21.292697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7882f2-2282-414a-9e11-bc2e72354c6a service swift] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:21.298545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:21.299170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:21.299452 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:21.299720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:21.301561 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:21.314244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:21.315877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:21.328680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1178/2352] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:19:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:21.340969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:21.344273 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:21.432493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d390469-47f1-40f0-a415-b95562108ca8 service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:21.438184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:21.438847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:21.439122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:21.439393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] PATH_INFO: `/v3/users/ffb896a1dae34a9dad20c23433855115` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:21.440609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ffb896a1dae34a9dad20c23433855115)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:21.442671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:21.501891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ffb896a1dae34a9dad20c23433855115'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.510671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ffb896a1dae34a9dad20c23433855115'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.511470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ffb896a1dae34a9dad20c23433855115'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.512159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ffb896a1dae34a9dad20c23433855115'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.522956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-70af8732-817e-4374-9b8b-101811b9a186 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ffb896a1dae34a9dad20c23433855115'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.550081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1175/2353] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:19:21 2022] DELETE /identity/v3/users/ffb896a1dae34a9dad20c23433855115 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:21.773660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:21.791844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0900651f-67b2-422c-96df-8fee9f7cd221 service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:21.865286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:21.866223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:21.866492 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:21.866759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] PATH_INFO: `/v3/projects/03571a9d7e1e43f1b5eef66eeae55844` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:21.868016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] RBAC: Authorizing `identity:delete_project(project_id=03571a9d7e1e43f1b5eef66eeae55844)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:21.869925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:21.930419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '03571a9d7e1e43f1b5eef66eeae55844', 'request_id': 'req-aeec86ad-d33c-44ee-af16-09890fac2aef'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.941983 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '03571a9d7e1e43f1b5eef66eeae55844', 'request_id': 'req-aeec86ad-d33c-44ee-af16-09890fac2aef'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.950392 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '03571a9d7e1e43f1b5eef66eeae55844', 'request_id': 'req-aeec86ad-d33c-44ee-af16-09890fac2aef'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.961679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] The token cache is being invalidate because project 03571a9d7e1e43f1b5eef66eeae55844 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:19:21.961916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.962785 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:19:21.964592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1179/2354] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:19:21 2022] DELETE /identity/v3/projects/03571a9d7e1e43f1b5eef66eeae55844 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:19:31.545525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=53,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:31.958530 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=57,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:34.700159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a386a2e7-16bb-4b82-b508-7fa79b17102f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:34.700706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a386a2e7-16bb-4b82-b508-7fa79b17102f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:34.700850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a386a2e7-16bb-4b82-b508-7fa79b17102f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:34.721341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a386a2e7-16bb-4b82-b508-7fa79b17102f None None] MFA Rules not processed for user `0df60784a8b3468ca19ec2cee1d929e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:19:34.722569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a386a2e7-16bb-4b82-b508-7fa79b17102f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:34.780733 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1176/2355] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:19:34 2022] POST /identity/v3/auth/tokens => generated 3270 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:19:41.239395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:19:41.241127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:41.324317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeec86ad-d33c-44ee-af16-09890fac2aef admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:19:41.329778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:19:41.330412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:19:41.330689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:19:41.330960 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:19:41.332687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:19:41.344697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:19:41.346303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:19:41.357664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1180/2356] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:19:41 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 119 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:19:44.773468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=25 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:19:51.352728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=34 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:09.457689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd0f5a-d225-42ad-8f72-9d59fd434f71 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:09.458253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd0f5a-d225-42ad-8f72-9d59fd434f71 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:09.458253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5bd0f5a-d225-42ad-8f72-9d59fd434f71 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:09.478108 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-c5bd0f5a-d225-42ad-8f72-9d59fd434f71 None None] MFA Rules not processed for user `a6111a16f95d4bc58f84d9f60f703d80`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:09.479294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c5bd0f5a-d225-42ad-8f72-9d59fd434f71 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:09.536299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1177/2357] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:20:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:19.529113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=25 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:22.639601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:22.651467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed55f4b-ae38-4999-94e9-dce88e85c642 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:22.658341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98460f8a-31ec-418a-809a-41206764227f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:22.658934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98460f8a-31ec-418a-809a-41206764227f service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:22.659207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98460f8a-31ec-418a-809a-41206764227f service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:22.659468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98460f8a-31ec-418a-809a-41206764227f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:22.661483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-98460f8a-31ec-418a-809a-41206764227f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:22.726782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98460f8a-31ec-418a-809a-41206764227f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:22.728372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98460f8a-31ec-418a-809a-41206764227f service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:22.737736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1181/2358] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:20:22 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:20:27.223749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:27.224189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:27.224300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:27.246366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] MFA Rules not processed for user `d0ed4854e0d449e58d09487fe8eedc2c`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:27.247509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:27.300937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1178/2359] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:20:27 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:32.733294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=28 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:35.897218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:35.897791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:35.897791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:35.918205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:20:35.919395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:35.978897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1182/2360] 10.222.0.61 () {58 vars in 967 bytes} [Thu Sep 29 02:20:35 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:20:36.031599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:36.033064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:36.106154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ac2037-ee0d-4b06-b52f-591955f2f688 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:36.111877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:36.112488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:36.112763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:36.113032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:36.121973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:36.123632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:36.131766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1179/2361] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:20:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:20:36.525690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:36.527220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:36.617948 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd4d4bc-6932-4971-83c2-9b76b5960043 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:36.622751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:36.623362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:36.623638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:36.623907 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:36.625769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:36.719875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:36.735862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:36.737349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:36.757531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1183/2362] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:20:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:20:43.543566 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:43.545890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:43.639616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:43.641079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:43.656725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637be2b3-e921-433c-9222-cfa5bf395ede service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:43.662475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:43.663513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:43.663944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:43.664403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] PATH_INFO: `/v3/users/92d1fb3d60734f5e9bde384f96c73d30` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:43.665945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] RBAC: Authorizing `identity:delete_user(user_id=92d1fb3d60734f5e9bde384f96c73d30)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:43.669106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:43.720986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:43.739979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '92d1fb3d60734f5e9bde384f96c73d30'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:43.748444 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '92d1fb3d60734f5e9bde384f96c73d30'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:43.749031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '92d1fb3d60734f5e9bde384f96c73d30'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:43.749743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '92d1fb3d60734f5e9bde384f96c73d30'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:43.759762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '92d1fb3d60734f5e9bde384f96c73d30'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:43.766686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d470e3-db38-4b3c-b01b-bfb50f0b812d service swift] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:43.772048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:43.772667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:43.772946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:43.773220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:43.775481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:43.785466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1184/2363] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:20:43 2022] DELETE /identity/v3/users/92d1fb3d60734f5e9bde384f96c73d30 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:20:43.790263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:43.791950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:43.803048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1180/2364] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:20:43 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 261 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:20:43.972602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:43.974129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:44.054776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec189e2f-2738-43b2-9a13-f80f3ddb070c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:44.060203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:44.060828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:44.061104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:44.061373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] PATH_INFO: `/v3/projects/30c052816d5544b38124d98d05b32798` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:44.062562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] RBAC: Authorizing `identity:delete_project(project_id=30c052816d5544b38124d98d05b32798)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:44.064471 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:44.115764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '30c052816d5544b38124d98d05b32798', 'request_id': 'req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.118499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:44.126303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '30c052816d5544b38124d98d05b32798', 'request_id': 'req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.133460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '30c052816d5544b38124d98d05b32798', 'request_id': 'req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.144155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] The token cache is being invalidate because project 30c052816d5544b38124d98d05b32798 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:20:44.144443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.145105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.146663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1185/2365] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:20:43 2022] DELETE /identity/v3/projects/30c052816d5544b38124d98d05b32798 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:20:44.150866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:44.161672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-528fa935-ffda-421b-bf4d-71a312f930a6 service placement] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:44.220423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:44.220966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:44.221236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:44.221502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] PATH_INFO: `/v3/users/3b65db35145749bfb82f0dc640f64a95` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:44.222621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] RBAC: Authorizing `identity:delete_user(user_id=3b65db35145749bfb82f0dc640f64a95)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:44.224569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:44.282334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3b65db35145749bfb82f0dc640f64a95'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.288844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3b65db35145749bfb82f0dc640f64a95'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.289482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3b65db35145749bfb82f0dc640f64a95'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.290117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3b65db35145749bfb82f0dc640f64a95'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.299862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3b65db35145749bfb82f0dc640f64a95'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.318736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1181/2366] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:20:44 2022] DELETE /identity/v3/users/3b65db35145749bfb82f0dc640f64a95 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:20:44.503142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:20:44.504752 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:20:44.518502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8ad0c04-0b97-412a-8372-1a9cea8baf1d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:20:44.584600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:20:44.585241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:20:44.585499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:20:44.585762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] PATH_INFO: `/v3/projects/75115cecb31d48abb6236ce3dc8cd829` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:20:44.587056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75115cecb31d48abb6236ce3dc8cd829)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:20:44.589364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:20:44.640770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75115cecb31d48abb6236ce3dc8cd829', 'request_id': 'req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.648885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75115cecb31d48abb6236ce3dc8cd829', 'request_id': 'req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.656594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75115cecb31d48abb6236ce3dc8cd829', 'request_id': 'req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.667958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] The token cache is being invalidate because project 75115cecb31d48abb6236ce3dc8cd829 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:20:44.668217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.668958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:20:44.670532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1186/2367] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:20:44 2022] DELETE /identity/v3/projects/75115cecb31d48abb6236ce3dc8cd829 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:20:54.315096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=65,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:20:54.664583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=10,keystone:SELECT=57,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:04.857472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:04.858905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:04.946522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0fc785-d4f4-4b40-9a51-ab912cdde0df admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:04.996083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:04.996686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:04.996953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:04.997230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:04.999512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:05.065749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:05.067169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:05.076946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1182/2368] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 220 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:08.283442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:08.284927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:08.301914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e172bd8-4ccc-465c-b40c-d4c51365a4c4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:08.306665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:08.307287 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:08.307588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:08.307860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] PATH_INFO: `/v3/users/0df60784a8b3468ca19ec2cee1d929e5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:08.309018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=0df60784a8b3468ca19ec2cee1d929e5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:08.311667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:08.366790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0df60784a8b3468ca19ec2cee1d929e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.374633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0df60784a8b3468ca19ec2cee1d929e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.375372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0df60784a8b3468ca19ec2cee1d929e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.376044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0df60784a8b3468ca19ec2cee1d929e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.386596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-afd9ab67-4882-4323-affa-5eefa70814ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0df60784a8b3468ca19ec2cee1d929e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.404988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1187/2369] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:08 2022] DELETE /identity/v3/users/0df60784a8b3468ca19ec2cee1d929e5 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:08.590757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:08.602514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69abb930-806e-42c4-a522-e84997c41cc6 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:08.649905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:08.663654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:08.664283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:08.664554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:08.664820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] PATH_INFO: `/v3/projects/8f88658c804e40229386a559fd2a1fa9` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:08.665938 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8f88658c804e40229386a559fd2a1fa9)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:08.667880 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:08.716394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8f88658c804e40229386a559fd2a1fa9', 'request_id': 'req-1aae9906-28e7-447b-b516-0a0e76f0ff52'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.725198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8f88658c804e40229386a559fd2a1fa9', 'request_id': 'req-1aae9906-28e7-447b-b516-0a0e76f0ff52'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.732101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8f88658c804e40229386a559fd2a1fa9', 'request_id': 'req-1aae9906-28e7-447b-b516-0a0e76f0ff52'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.743972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] The token cache is being invalidate because project 8f88658c804e40229386a559fd2a1fa9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:08.744286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.745067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1aae9906-28e7-447b-b516-0a0e76f0ff52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:08.747434 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1183/2370] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:08 2022] DELETE /identity/v3/projects/8f88658c804e40229386a559fd2a1fa9 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:08.754685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cf585a-a060-40cf-9648-8934495002de None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:08.754995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cf585a-a060-40cf-9648-8934495002de None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:08.755310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63cf585a-a060-40cf-9648-8934495002de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:08.824998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-63cf585a-a060-40cf-9648-8934495002de None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:08.826278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-63cf585a-a060-40cf-9648-8934495002de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:08.887036 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1188/2371] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:08.895495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b128aa35-94ea-4564-ab77-da7864830d02 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:08.895799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b128aa35-94ea-4564-ab77-da7864830d02 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:08.896067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b128aa35-94ea-4564-ab77-da7864830d02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:08.917172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b128aa35-94ea-4564-ab77-da7864830d02 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:08.918406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b128aa35-94ea-4564-ab77-da7864830d02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:08.928578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1184/2372] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:08.933080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cf585a-a060-40cf-9648-8934495002de None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:08.944693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63cf585a-a060-40cf-9648-8934495002de None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:08.950809 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:08.951407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:08.951708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:08.951976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:08.952620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:08.952921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:08.954566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:08.967377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1189/2373] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:08.971974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b128aa35-94ea-4564-ab77-da7864830d02 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:08.981411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b128aa35-94ea-4564-ab77-da7864830d02 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:08.986950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:08.987540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:08.987808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:08.988070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:08.988691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:08.990643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:09.021158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1185/2374] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:08 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:09.025923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:09.039088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17699bfb-104e-4095-bc87-113745567aa1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:09.067621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:09.108044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:09.108645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:09.108921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:09.109192 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:09.109834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:09.112093 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:09.137497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1190/2375] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:09 2022] POST /identity/v3/users => generated 293 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:21:09.143658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:09.158849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d140290-56d4-4afc-ba49-e3056b82b117 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:09.163815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:09.164407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:09.164722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:09.164969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:09.165599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:09.167278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:09.177600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1186/2376] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:21:09.181718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:09.193013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1c817a-2860-4801-b9f7-244c8c9e8479 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:09.198886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9de79abc-6d2e-4179-8361-50651f94a0df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:09.199475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de79abc-6d2e-4179-8361-50651f94a0df admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:09.199749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de79abc-6d2e-4179-8361-50651f94a0df admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:09.200013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9de79abc-6d2e-4179-8361-50651f94a0df admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:09.200644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de79abc-6d2e-4179-8361-50651f94a0df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:09.202266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9de79abc-6d2e-4179-8361-50651f94a0df admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:09.213221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1191/2377] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:09.217493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:09.228280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca69bac6-f66e-4394-93b6-bb27761e397b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:09.233346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:09.233985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:09.234293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:09.234561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] PATH_INFO: `/v3/projects/f6207fc27f8546028eced464da11be94/users/a86492e006284f44af99d08ef7e74559/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:09.249680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f6207fc27f8546028eced464da11be94, user_id=a86492e006284f44af99d08ef7e74559, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:09.252695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:09.270174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1187/2378] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:09 2022] PUT /identity/v3/projects/f6207fc27f8546028eced464da11be94/users/a86492e006284f44af99d08ef7e74559/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:09.281703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d1c4b7-b214-4161-bbf4-d0e0a4bf35da None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:09.281976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d1c4b7-b214-4161-bbf4-d0e0a4bf35da None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:09.282273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63d1c4b7-b214-4161-bbf4-d0e0a4bf35da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:09.312519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-63d1c4b7-b214-4161-bbf4-d0e0a4bf35da None None] MFA Rules not processed for user `a86492e006284f44af99d08ef7e74559`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:09.314139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-63d1c4b7-b214-4161-bbf4-d0e0a4bf35da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:09.387425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1192/2379] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:09 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:21:09.400588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:09.402482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:09.477074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:09.512351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda9aeae-14cb-4e65-b9d8-d87326108903 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:09.518146 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f6e54f1-507c-4210-9268-3a436372e668 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:09.518920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6e54f1-507c-4210-9268-3a436372e668 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:09.519195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6e54f1-507c-4210-9268-3a436372e668 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:09.519462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f6e54f1-507c-4210-9268-3a436372e668 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:09.527327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6e54f1-507c-4210-9268-3a436372e668 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:09.528962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f6e54f1-507c-4210-9268-3a436372e668 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:09.538098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1188/2380] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:09 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:19.042245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d1c4b7-b214-4161-bbf4-d0e0a4bf35da None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:19.107400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63d1c4b7-b214-4161-bbf4-d0e0a4bf35da None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:19.112958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:19.113595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.113867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.114173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] PATH_INFO: `/v3/users/a86492e006284f44af99d08ef7e74559` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.115347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a86492e006284f44af99d08ef7e74559)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:19.117289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:19.164276 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:19.171987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a86492e006284f44af99d08ef7e74559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.179847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a86492e006284f44af99d08ef7e74559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.180489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a86492e006284f44af99d08ef7e74559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.181077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a86492e006284f44af99d08ef7e74559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.189377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a86492e006284f44af99d08ef7e74559'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.206180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1193/2381] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:19 2022] DELETE /identity/v3/users/a86492e006284f44af99d08ef7e74559 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:19.210824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.211323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.211432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.231360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:19.232539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:19.293829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1189/2382] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:19.303964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:19.305551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:19.371219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-278f4b3f-a79f-4533-bb93-70f25e32c2f1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:19.371219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aef3a5f0-9555-4635-9c9b-7c817d7a854c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:19.371219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef3a5f0-9555-4635-9c9b-7c817d7a854c service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.371219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef3a5f0-9555-4635-9c9b-7c817d7a854c service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.371798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aef3a5f0-9555-4635-9c9b-7c817d7a854c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.381311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef3a5f0-9555-4635-9c9b-7c817d7a854c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:19.382775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aef3a5f0-9555-4635-9c9b-7c817d7a854c service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:19.393450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1194/2383] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:19.676064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:19.677612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:19.689843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e8ef64-0ba9-4e75-8687-7e0b61d4a3ba None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:19.695087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:19.695723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.696015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.696279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] PATH_INFO: `/v3/projects/f6207fc27f8546028eced464da11be94` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.697471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f6207fc27f8546028eced464da11be94)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:19.699767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:19.752616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f6207fc27f8546028eced464da11be94', 'request_id': 'req-97d80484-636d-42ec-95f3-d7e42ca18a29'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.761563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f6207fc27f8546028eced464da11be94', 'request_id': 'req-97d80484-636d-42ec-95f3-d7e42ca18a29'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.767995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f6207fc27f8546028eced464da11be94', 'request_id': 'req-97d80484-636d-42ec-95f3-d7e42ca18a29'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.778249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] The token cache is being invalidate because project f6207fc27f8546028eced464da11be94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:19.778534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.779211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-97d80484-636d-42ec-95f3-d7e42ca18a29 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:19.780736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1190/2384] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:19 2022] DELETE /identity/v3/projects/f6207fc27f8546028eced464da11be94 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:19.788111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a5d2ce-c4e0-45c1-aec2-b1b9f4404213 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.788465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a5d2ce-c4e0-45c1-aec2-b1b9f4404213 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.788829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09a5d2ce-c4e0-45c1-aec2-b1b9f4404213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.812732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-09a5d2ce-c4e0-45c1-aec2-b1b9f4404213 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:19.813994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-09a5d2ce-c4e0-45c1-aec2-b1b9f4404213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:19.879462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1195/2385] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:19.887906 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4f70a8-2cf6-492c-b1b2-39a6e452f585 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.888184 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4f70a8-2cf6-492c-b1b2-39a6e452f585 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.888455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4f70a8-2cf6-492c-b1b2-39a6e452f585 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.908848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-6f4f70a8-2cf6-492c-b1b2-39a6e452f585 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:19.910101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6f4f70a8-2cf6-492c-b1b2-39a6e452f585 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:19.918848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1191/2386] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:19.923178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a5d2ce-c4e0-45c1-aec2-b1b9f4404213 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:19.933321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09a5d2ce-c4e0-45c1-aec2-b1b9f4404213 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:19.938999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:19.939595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.939867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.940136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.940781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:19.941059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:19.942718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:19.954866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1196/2387] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:19.959145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4f70a8-2cf6-492c-b1b2-39a6e452f585 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:19.968729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4f70a8-2cf6-492c-b1b2-39a6e452f585 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:19.974611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:19.975234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:19.975504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:19.975767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:19.976403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:19.978339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:20.009432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1192/2388] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:19 2022] POST /identity/v3/projects => generated 373 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:21:20.014020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:20.024379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a9f45be-859a-4f52-a15b-0422836765b9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:20.073734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=5,keystone:SELECT=94,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:20.088944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:20.089591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.089867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.090167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.090800 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:20.092717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:20.116152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1197/2389] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:20 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:20.120745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:20.132172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da3338f8-c6a3-40af-af56-824dc1b28eca admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:20.137446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:20.138028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.138345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.138619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.139254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:20.140887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:20.151603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1193/2390] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:20.155816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:20.165831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5acf68-200f-4080-ad1a-f1311f8ed0fe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:20.170972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a12ac5a-bbdf-4bae-8de3-f1066050f49b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:20.171606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a12ac5a-bbdf-4bae-8de3-f1066050f49b admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.171884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a12ac5a-bbdf-4bae-8de3-f1066050f49b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.172158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a12ac5a-bbdf-4bae-8de3-f1066050f49b admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.172804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a12ac5a-bbdf-4bae-8de3-f1066050f49b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:20.174510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a12ac5a-bbdf-4bae-8de3-f1066050f49b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:20.185937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1198/2391] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:20 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:20.190401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:20.201080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d367c-d9ac-4e44-a683-50ec466db772 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:20.207056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:20.207709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.207982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.208257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] PATH_INFO: `/v3/projects/dc6e32ff424f435d89bef15929d19218/users/ec579cfaecca4fb18679e8b7ea07b1e5/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.221525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:20.226033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc6e32ff424f435d89bef15929d19218, user_id=ec579cfaecca4fb18679e8b7ea07b1e5, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:20.229142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:20.243718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1194/2392] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:20 2022] PUT /identity/v3/projects/dc6e32ff424f435d89bef15929d19218/users/ec579cfaecca4fb18679e8b7ea07b1e5/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:20.254795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c45ec-2faa-4c56-a67a-ee959079a968 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.255074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c45ec-2faa-4c56-a67a-ee959079a968 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.255344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8c45ec-2faa-4c56-a67a-ee959079a968 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.284625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5c8c45ec-2faa-4c56-a67a-ee959079a968 None None] MFA Rules not processed for user `ec579cfaecca4fb18679e8b7ea07b1e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:20.285746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5c8c45ec-2faa-4c56-a67a-ee959079a968 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:20.353116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1199/2393] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:20 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:21:20.364692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:20.366482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:20.453675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6803992c-4aa1-4d72-a4b0-ad9303a26fc6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:20.459754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcbae7b-a4fa-457b-8613-a34cf2e89a79 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:20.460372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcbae7b-a4fa-457b-8613-a34cf2e89a79 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.460652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcbae7b-a4fa-457b-8613-a34cf2e89a79 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.460926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcbae7b-a4fa-457b-8613-a34cf2e89a79 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.469415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcbae7b-a4fa-457b-8613-a34cf2e89a79 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:20.471045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcbae7b-a4fa-457b-8613-a34cf2e89a79 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:20.479898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1195/2394] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:20 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:20.701340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c45ec-2faa-4c56-a67a-ee959079a968 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:20.770838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c8c45ec-2faa-4c56-a67a-ee959079a968 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:20.775870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:20.776487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.776795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.777088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] PATH_INFO: `/v3/users/ec579cfaecca4fb18679e8b7ea07b1e5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.778261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ec579cfaecca4fb18679e8b7ea07b1e5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:20.780172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:20.834250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ec579cfaecca4fb18679e8b7ea07b1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:20.841340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ec579cfaecca4fb18679e8b7ea07b1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:20.841991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ec579cfaecca4fb18679e8b7ea07b1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:20.842621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ec579cfaecca4fb18679e8b7ea07b1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:20.844271 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:20.852729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ec579cfaecca4fb18679e8b7ea07b1e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:20.872321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1200/2395] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:20 2022] DELETE /identity/v3/users/ec579cfaecca4fb18679e8b7ea07b1e5 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:20.877299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:20.877587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:20.877855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:20.897745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:20.898980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:20.960675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1196/2396] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:20.970850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:20.972379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:21.030225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6152c4bd-0a99-4458-bdad-819a73321669 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.034342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-617ea380-145c-4150-9201-d4fbb9696f8b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.034938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ea380-145c-4150-9201-d4fbb9696f8b service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.035213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ea380-145c-4150-9201-d4fbb9696f8b service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.035488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-617ea380-145c-4150-9201-d4fbb9696f8b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.045172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ea380-145c-4150-9201-d4fbb9696f8b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.046643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-617ea380-145c-4150-9201-d4fbb9696f8b service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.056180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1201/2397] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:21.332555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:21.334117 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:21.346595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dff8aa46-9ccc-4caa-bff9-189c40cae7ab None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.351921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.352551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.352800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.353066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] PATH_INFO: `/v3/projects/dc6e32ff424f435d89bef15929d19218` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.354340 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc6e32ff424f435d89bef15929d19218)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.356408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.407082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc6e32ff424f435d89bef15929d19218', 'request_id': 'req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:21.415865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc6e32ff424f435d89bef15929d19218', 'request_id': 'req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:21.425746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc6e32ff424f435d89bef15929d19218', 'request_id': 'req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:21.435652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:21.440023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] The token cache is being invalidate because project dc6e32ff424f435d89bef15929d19218 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:21.440315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:21.441431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c88d35d8-ee20-4810-90ea-8fcdc35e9c84 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:21.443031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1197/2398] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:21 2022] DELETE /identity/v3/projects/dc6e32ff424f435d89bef15929d19218 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:21.448354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41105201-e3ef-4566-b463-e235630c31c1 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.448928 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41105201-e3ef-4566-b463-e235630c31c1 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.449100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41105201-e3ef-4566-b463-e235630c31c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.470017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-41105201-e3ef-4566-b463-e235630c31c1 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:21.471258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-41105201-e3ef-4566-b463-e235630c31c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:21.535635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1202/2399] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:21.546080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6d5ffa-025f-44e1-9b4f-cc15ec789c7b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.546369 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6d5ffa-025f-44e1-9b4f-cc15ec789c7b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.546638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf6d5ffa-025f-44e1-9b4f-cc15ec789c7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.567536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-bf6d5ffa-025f-44e1-9b4f-cc15ec789c7b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:21.568744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bf6d5ffa-025f-44e1-9b4f-cc15ec789c7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:21.577534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1198/2400] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:21.581896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41105201-e3ef-4566-b463-e235630c31c1 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:21.591793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41105201-e3ef-4566-b463-e235630c31c1 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.596615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.597208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.597484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.597754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.598456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:21.598753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.600353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.612247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1203/2401] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:21.616666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6d5ffa-025f-44e1-9b4f-cc15ec789c7b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:21.626969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf6d5ffa-025f-44e1-9b4f-cc15ec789c7b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.632314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.632915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.633189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.633460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.634109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.636034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.665651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1199/2402] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:21 2022] POST /identity/v3/projects => generated 363 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:21.669916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:21.679946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fa0064-6e85-491d-ac7f-24a1ea0124ae admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.741223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.741847 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.742148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.742416 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.743043 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.744959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.755554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:INSERT=3,keystone:DELETE=3,keystone:SELECT=94 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:21.769879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1204/2403] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:21 2022] POST /identity/v3/users => generated 282 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:21.774230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:21.785736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa7242b-43fa-401e-b24e-17e017f28308 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.791182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.791784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.792059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.792335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.792975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.794797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.805344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1200/2404] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:21.809529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:21.820340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8c84f9-74de-4416-92b3-a9f4f1ead3a4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.826750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181b8f59-84fc-472f-856d-2873746e440b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.827353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181b8f59-84fc-472f-856d-2873746e440b admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.827623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181b8f59-84fc-472f-856d-2873746e440b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.827895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181b8f59-84fc-472f-856d-2873746e440b admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.828544 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181b8f59-84fc-472f-856d-2873746e440b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.830179 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181b8f59-84fc-472f-856d-2873746e440b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.841499 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1205/2405] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:21:21.846872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:21.857410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ed72a45-517e-4352-acc3-5ed2bf7d890c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:21.862396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:21.863044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.863318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.863609 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] PATH_INFO: `/v3/projects/586d7614e7cf4c0fbd54465394217e66/users/a2bddac14e8244399ce7976d465eb502/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.879042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] RBAC: Authorizing `identity:create_grant(project_id=586d7614e7cf4c0fbd54465394217e66, user_id=a2bddac14e8244399ce7976d465eb502, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:21.881707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:21.897257 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1201/2406] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:21 2022] PUT /identity/v3/projects/586d7614e7cf4c0fbd54465394217e66/users/a2bddac14e8244399ce7976d465eb502/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:21:21.908679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f9d30-1c9a-489e-bbda-38a490acff0e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:21.909054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f9d30-1c9a-489e-bbda-38a490acff0e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:21.909432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-533f9d30-1c9a-489e-bbda-38a490acff0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:21.946925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-533f9d30-1c9a-489e-bbda-38a490acff0e None None] MFA Rules not processed for user `a2bddac14e8244399ce7976d465eb502`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:21.948052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-533f9d30-1c9a-489e-bbda-38a490acff0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:22.014418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1206/2407] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:21 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:21:22.019079 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.081291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7212b497-7d2e-47e6-b7c0-567afd68d544 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.086022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.086719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.086980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.087271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.088701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.089748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.120252 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1202/2408] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:22 2022] POST /identity/v3/projects => generated 365 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:22.125554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f9d30-1c9a-489e-bbda-38a490acff0e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.136985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-533f9d30-1c9a-489e-bbda-38a490acff0e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.199184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.199707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.200099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.200470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.201100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.203068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.230568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1207/2409] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:22 2022] POST /identity/v3/users => generated 289 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:22.234647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.244855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f4285ee-3d6d-4226-aabd-59f35fa143c3 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.250618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.251208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.251488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.251750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.252385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.254016 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.266150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1203/2410] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:22 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:22.270510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.281166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3898570d-dc22-4a72-8bd7-cd4d9834f1f8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.291902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.292620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.292931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.293250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] PATH_INFO: `/v3/projects/969fcc330a3f4fdc85bdfe12f631b73e/users/88bbf45fb8dc4520b239834d9ae9317b/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.309485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=969fcc330a3f4fdc85bdfe12f631b73e, user_id=88bbf45fb8dc4520b239834d9ae9317b, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.312178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.326341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1208/2411] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:22 2022] PUT /identity/v3/projects/969fcc330a3f4fdc85bdfe12f631b73e/users/88bbf45fb8dc4520b239834d9ae9317b/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:22.330638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.341498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-079612bb-da63-4a62-811a-633e4a0b7ffd admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.404270 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4500610f-e1ca-4197-8b97-5e057b6e309e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.404873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4500610f-e1ca-4197-8b97-5e057b6e309e admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.405147 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4500610f-e1ca-4197-8b97-5e057b6e309e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.405398 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4500610f-e1ca-4197-8b97-5e057b6e309e admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.406063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4500610f-e1ca-4197-8b97-5e057b6e309e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.407700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4500610f-e1ca-4197-8b97-5e057b6e309e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.417851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1204/2412] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:22 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:22.422465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.433205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1de8c2-8989-47af-a4cf-d55eb1d6efe3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.438327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.438978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.439284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.439556 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] PATH_INFO: `/v3/projects/969fcc330a3f4fdc85bdfe12f631b73e/users/88bbf45fb8dc4520b239834d9ae9317b/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.441601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] RBAC: Authorizing `identity:create_grant(project_id=969fcc330a3f4fdc85bdfe12f631b73e, user_id=88bbf45fb8dc4520b239834d9ae9317b, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.444306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.455028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1209/2413] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:22 2022] PUT /identity/v3/projects/969fcc330a3f4fdc85bdfe12f631b73e/users/88bbf45fb8dc4520b239834d9ae9317b/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:21:22.467305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f1bf9-9fee-4257-88af-3307beb4a631 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.467643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f1bf9-9fee-4257-88af-3307beb4a631 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.467927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5f1bf9-9fee-4257-88af-3307beb4a631 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.486174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:22.502824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-4a5f1bf9-9fee-4257-88af-3307beb4a631 None None] MFA Rules not processed for user `88bbf45fb8dc4520b239834d9ae9317b`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:22.504034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4a5f1bf9-9fee-4257-88af-3307beb4a631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:22.576165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1205/2414] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:22 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:22.588154 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.590378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:22.621479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:22.677293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37c1982-e5b5-46b2-8c56-1f898643ad8b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.681177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.681770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.682064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.682335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.692689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.694127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.703520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1210/2415] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:22 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:22.791997 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f1bf9-9fee-4257-88af-3307beb4a631 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:22.803133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5f1bf9-9fee-4257-88af-3307beb4a631 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:22.806745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40d126f-7fac-4dfc-a144-18ea4938d80d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:22.807366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40d126f-7fac-4dfc-a144-18ea4938d80d service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:22.807638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40d126f-7fac-4dfc-a144-18ea4938d80d service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:22.807904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40d126f-7fac-4dfc-a144-18ea4938d80d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:22.864483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40d126f-7fac-4dfc-a144-18ea4938d80d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:22.865916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40d126f-7fac-4dfc-a144-18ea4938d80d service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:22.875769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1206/2416] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:22 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:21:25.738654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:25.814099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aedecd7f-2114-4442-8239-e72fa04492b7 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:25.819687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:25.820309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:25.820585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:25.820856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] PATH_INFO: `/v3/users/a2bddac14e8244399ce7976d465eb502` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:25.822017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2bddac14e8244399ce7976d465eb502)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:25.823966 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:25.874703 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2bddac14e8244399ce7976d465eb502'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.881671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2bddac14e8244399ce7976d465eb502'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.882394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2bddac14e8244399ce7976d465eb502'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.883007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2bddac14e8244399ce7976d465eb502'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.894971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2bddac14e8244399ce7976d465eb502'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:25.917807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1211/2417] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:25 2022] DELETE /identity/v3/users/a2bddac14e8244399ce7976d465eb502 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:25.922729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:25.923274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:25.923274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:25.943094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:25.944253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:26.005070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1207/2418] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:26.015232 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:26.016831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:26.049860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:26.078319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-351335eb-6461-440c-ba1d-ba3f5f4c5393 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:26.082261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:26.083068 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:26.083183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:26.083403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:26.093124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:26.094595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:26.104384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1212/2419] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:26.386132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:26.387576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:26.399798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63275f4c-87d5-464d-a5e3-60e86d70d35f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:26.405413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:26.406084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:26.406365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:26.406632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] PATH_INFO: `/v3/projects/586d7614e7cf4c0fbd54465394217e66` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:26.407783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] RBAC: Authorizing `identity:delete_project(project_id=586d7614e7cf4c0fbd54465394217e66)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:26.409732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:26.442507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:26.459393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '586d7614e7cf4c0fbd54465394217e66', 'request_id': 'req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.468021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '586d7614e7cf4c0fbd54465394217e66', 'request_id': 'req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.474507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '586d7614e7cf4c0fbd54465394217e66', 'request_id': 'req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.485323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] The token cache is being invalidate because project 586d7614e7cf4c0fbd54465394217e66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:26.485601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.486274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.487778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1208/2420] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:26 2022] DELETE /identity/v3/projects/586d7614e7cf4c0fbd54465394217e66 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:26.492039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:26.502868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f9ec0e-a102-41cf-b007-babdd64956e4 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:26.571131 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:26.571743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:26.572015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:26.572282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] PATH_INFO: `/v3/users/88bbf45fb8dc4520b239834d9ae9317b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:26.573433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] RBAC: Authorizing `identity:delete_user(user_id=88bbf45fb8dc4520b239834d9ae9317b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:26.575417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:26.622513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88bbf45fb8dc4520b239834d9ae9317b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.631066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88bbf45fb8dc4520b239834d9ae9317b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.631643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88bbf45fb8dc4520b239834d9ae9317b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.632238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88bbf45fb8dc4520b239834d9ae9317b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.641698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2a813e8d-9438-4863-a85a-dfa0b5bb8e4e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88bbf45fb8dc4520b239834d9ae9317b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:26.659665 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1213/2421] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:26 2022] DELETE /identity/v3/users/88bbf45fb8dc4520b239834d9ae9317b => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:21:26.932392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:26.933907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:26.949476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9baaa4ab-f0a8-4202-894a-6701d9c42f7e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.009295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.009931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.010299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.010568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] PATH_INFO: `/v3/projects/969fcc330a3f4fdc85bdfe12f631b73e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.011659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] RBAC: Authorizing `identity:delete_project(project_id=969fcc330a3f4fdc85bdfe12f631b73e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.013675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.060432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '969fcc330a3f4fdc85bdfe12f631b73e', 'request_id': 'req-086baf25-ef80-4de4-a2cd-a6102350a252'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:27.069155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '969fcc330a3f4fdc85bdfe12f631b73e', 'request_id': 'req-086baf25-ef80-4de4-a2cd-a6102350a252'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:27.075547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '969fcc330a3f4fdc85bdfe12f631b73e', 'request_id': 'req-086baf25-ef80-4de4-a2cd-a6102350a252'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:27.086322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] The token cache is being invalidate because project 969fcc330a3f4fdc85bdfe12f631b73e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:27.086606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:27.087258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-086baf25-ef80-4de4-a2cd-a6102350a252 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:27.088740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1209/2422] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:26 2022] DELETE /identity/v3/projects/969fcc330a3f4fdc85bdfe12f631b73e => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:27.094437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c1c06f-73e4-469c-8cc9-ed2511c2874a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.094920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c1c06f-73e4-469c-8cc9-ed2511c2874a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.095014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c1c06f-73e4-469c-8cc9-ed2511c2874a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.115375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c0c1c06f-73e4-469c-8cc9-ed2511c2874a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:27.116613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c0c1c06f-73e4-469c-8cc9-ed2511c2874a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:27.166521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:27.187486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1214/2423] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:27.195167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90056d69-2cbb-41e3-97b8-f5cb4e6b5a79 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.195448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90056d69-2cbb-41e3-97b8-f5cb4e6b5a79 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.195749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90056d69-2cbb-41e3-97b8-f5cb4e6b5a79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.216715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-90056d69-2cbb-41e3-97b8-f5cb4e6b5a79 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:27.217909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-90056d69-2cbb-41e3-97b8-f5cb4e6b5a79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:27.227521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1210/2424] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:27.232336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c1c06f-73e4-469c-8cc9-ed2511c2874a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.242562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c1c06f-73e4-469c-8cc9-ed2511c2874a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.248437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.249370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.249650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.249970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.250685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:27.250988 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.252633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.265266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1215/2425] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:27.270272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90056d69-2cbb-41e3-97b8-f5cb4e6b5a79 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.283367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90056d69-2cbb-41e3-97b8-f5cb4e6b5a79 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.288980 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.289576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.289849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.290137 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.290774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.292717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.324352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1211/2426] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:27 2022] POST /identity/v3/projects => generated 369 bytes in 55 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:27.329071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.339347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921e3edb-644b-40a0-91b7-4f2c65dec0fe admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.407168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.407774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.408040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.408304 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.408922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.410893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.437058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1216/2427] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:27 2022] POST /identity/v3/users => generated 285 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:27.441378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.451550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7edeff2e-9516-4a05-8ad5-794760a077a9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.456496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.457103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.457379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.457650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.458308 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.459927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.469960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1212/2428] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:27 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:27.473885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.484296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac3ec56-b409-4bcf-9d3d-693f2f00c3d8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.489787 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62162c54-ade7-4636-b147-33b3465b85c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.490407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62162c54-ade7-4636-b147-33b3465b85c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.490681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62162c54-ade7-4636-b147-33b3465b85c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.490946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62162c54-ade7-4636-b147-33b3465b85c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.491582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62162c54-ade7-4636-b147-33b3465b85c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.493215 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62162c54-ade7-4636-b147-33b3465b85c8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.505023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1217/2429] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:27.509247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.516173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=13,keystone:SELECT=84,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:27.520072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4f64b2-8507-4bec-b6e3-a43810c1cd59 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.526982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.527681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.527960 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.528231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] PATH_INFO: `/v3/projects/d0bb74f9f34c4fa2a5a23b69eb43a30b/users/54a1d131218b44feb67782b33fb23387/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.546242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d0bb74f9f34c4fa2a5a23b69eb43a30b, user_id=54a1d131218b44feb67782b33fb23387, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.548968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.564404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1213/2430] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:27 2022] PUT /identity/v3/projects/d0bb74f9f34c4fa2a5a23b69eb43a30b/users/54a1d131218b44feb67782b33fb23387/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:27.574867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bdd4f-0159-4bcd-9b12-bee9a13aa373 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.575158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bdd4f-0159-4bcd-9b12-bee9a13aa373 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.575455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97bdd4f-0159-4bcd-9b12-bee9a13aa373 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.610212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c97bdd4f-0159-4bcd-9b12-bee9a13aa373 None None] MFA Rules not processed for user `54a1d131218b44feb67782b33fb23387`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:27.611746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c97bdd4f-0159-4bcd-9b12-bee9a13aa373 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:27.683875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1218/2431] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:27 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:27.688647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.756552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42911dbc-bf77-414a-aea1-2215e9a7ee13 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.762220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.762825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.763096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.763439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.764101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.765986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.794958 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1214/2432] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:27 2022] POST /identity/v3/projects => generated 367 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:27.799455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bdd4f-0159-4bcd-9b12-bee9a13aa373 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.810854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97bdd4f-0159-4bcd-9b12-bee9a13aa373 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.856214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:27.878540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.879147 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.879425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.879689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.880336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.882311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.908319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1219/2433] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:27 2022] POST /identity/v3/users => generated 290 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:27.912684 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.924234 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63df4730-209f-4ee3-8966-c4f2eb909086 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.929286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.929884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.930192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.930461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.931097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.932703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:27.944299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1215/2434] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:27 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:27.949189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:27.960030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c7254f1-370b-4d16-8226-410e2de61549 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:27.965116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:27.965764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:27.966064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:27.966341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] PATH_INFO: `/v3/projects/05740a7d1d0c40bbbe3408de006c3d57/users/611f4ae650e941c18e6b2790ae683b49/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:27.982639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05740a7d1d0c40bbbe3408de006c3d57, user_id=611f4ae650e941c18e6b2790ae683b49, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:27.986834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:28.004438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1220/2435] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:27 2022] PUT /identity/v3/projects/05740a7d1d0c40bbbe3408de006c3d57/users/611f4ae650e941c18e6b2790ae683b49/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:28.008594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:28.019004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b589e33-7a89-42b9-bb67-278709975708 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:28.085159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfe04356-d37a-454d-ad82-a3c9091604a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:28.085774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe04356-d37a-454d-ad82-a3c9091604a9 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:28.086052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe04356-d37a-454d-ad82-a3c9091604a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:28.086347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfe04356-d37a-454d-ad82-a3c9091604a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:28.086992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe04356-d37a-454d-ad82-a3c9091604a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:28.088644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfe04356-d37a-454d-ad82-a3c9091604a9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:28.098547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1216/2436] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:28 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:28.102972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:28.114410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf4feb0-1c03-4f4e-af66-3bf88e06e950 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:28.119629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:28.120273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:28.120547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:28.120814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] PATH_INFO: `/v3/projects/05740a7d1d0c40bbbe3408de006c3d57/users/611f4ae650e941c18e6b2790ae683b49/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:28.122877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05740a7d1d0c40bbbe3408de006c3d57, user_id=611f4ae650e941c18e6b2790ae683b49, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:28.126555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:28.137475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1221/2437] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:28 2022] PUT /identity/v3/projects/05740a7d1d0c40bbbe3408de006c3d57/users/611f4ae650e941c18e6b2790ae683b49/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:28.149648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:28.149922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:28.150237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:28.179893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] MFA Rules not processed for user `611f4ae650e941c18e6b2790ae683b49`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:28.181106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:28.221358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:28.259463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1217/2438] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:28 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:28.271641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:28.274220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:28.343262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701321e7-bf1e-4ecc-a40d-85167b3face5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:28.347342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:28.347941 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:28.348172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:28.348473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:28.358554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:28.359888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:28.369481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1222/2439] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:28 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:28.425655 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:28.427267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:28.440436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5205f2b-35a5-42e9-9a48-e7fd81704729 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:28.445030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67b1465-c7ca-4408-b9a0-10747a69a271 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:28.445635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b1465-c7ca-4408-b9a0-10747a69a271 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:28.445907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b1465-c7ca-4408-b9a0-10747a69a271 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:28.446222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67b1465-c7ca-4408-b9a0-10747a69a271 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:28.506289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67b1465-c7ca-4408-b9a0-10747a69a271 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:28.507744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67b1465-c7ca-4408-b9a0-10747a69a271 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:28.517305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1218/2440] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:28 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:31.895328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:31.963223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bc938e6-de6c-4b42-bf14-e6ac0bab3bdf service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:31.968500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:31.969136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:31.969415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:31.969709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] PATH_INFO: `/v3/users/54a1d131218b44feb67782b33fb23387` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:31.970937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] RBAC: Authorizing `identity:delete_user(user_id=54a1d131218b44feb67782b33fb23387)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:31.972880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:32.024388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:INSERT=5,keystone:DELETE=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:32.031258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '54a1d131218b44feb67782b33fb23387'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.039823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '54a1d131218b44feb67782b33fb23387'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.040697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '54a1d131218b44feb67782b33fb23387'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.041442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '54a1d131218b44feb67782b33fb23387'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.050497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '54a1d131218b44feb67782b33fb23387'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.067972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1223/2441] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:31 2022] DELETE /identity/v3/users/54a1d131218b44feb67782b33fb23387 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:32.072521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:32.073055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:32.073055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:32.093255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:32.094415 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:32.156766 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1219/2442] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:32 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:21:32.166886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:32.169650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:32.238724 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6310d531-a2ff-4250-8353-9910acfcc071 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:32.242571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:32.243177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:32.243448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:32.243722 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:32.253862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:32.255320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:32.264994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1224/2443] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:32.548865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:32.550455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:32.562817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37980bc-8f16-4f46-a60b-35611b1b3c40 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:32.568824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:32.569483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:32.569758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:32.570056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] PATH_INFO: `/v3/projects/d0bb74f9f34c4fa2a5a23b69eb43a30b` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:32.571215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d0bb74f9f34c4fa2a5a23b69eb43a30b)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:32.573391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:32.623152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd0bb74f9f34c4fa2a5a23b69eb43a30b', 'request_id': 'req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.635049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd0bb74f9f34c4fa2a5a23b69eb43a30b', 'request_id': 'req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.644327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd0bb74f9f34c4fa2a5a23b69eb43a30b', 'request_id': 'req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.655437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:32.661481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] The token cache is being invalidate because project d0bb74f9f34c4fa2a5a23b69eb43a30b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:32.661950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.662805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.664786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1220/2444] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:32 2022] DELETE /identity/v3/projects/d0bb74f9f34c4fa2a5a23b69eb43a30b => generated 0 bytes in 117 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:32.669008 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:32.679129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca8a5aa8-467d-4e83-acae-f9dcd200c047 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:32.738920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:32.739558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:32.739836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:32.740106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] PATH_INFO: `/v3/users/611f4ae650e941c18e6b2790ae683b49` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:32.741247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] RBAC: Authorizing `identity:delete_user(user_id=611f4ae650e941c18e6b2790ae683b49)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:32.743245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:32.796813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '611f4ae650e941c18e6b2790ae683b49'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.804048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '611f4ae650e941c18e6b2790ae683b49'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.804855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '611f4ae650e941c18e6b2790ae683b49'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.805465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '611f4ae650e941c18e6b2790ae683b49'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.813466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-10e5fa72-c499-466a-896c-57e398039234 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '611f4ae650e941c18e6b2790ae683b49'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:32.814734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:32.834028 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1225/2445] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:32 2022] DELETE /identity/v3/users/611f4ae650e941c18e6b2790ae683b49 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:33.124864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.127133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:33.141068 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff326b05-2ef3-4e04-8b8e-334ce5fb0c89 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:33.212406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:33.213956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.214438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.214851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] PATH_INFO: `/v3/projects/05740a7d1d0c40bbbe3408de006c3d57` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.216292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] RBAC: Authorizing `identity:delete_project(project_id=05740a7d1d0c40bbbe3408de006c3d57)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:33.218914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:33.306497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05740a7d1d0c40bbbe3408de006c3d57', 'request_id': 'req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:33.319937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05740a7d1d0c40bbbe3408de006c3d57', 'request_id': 'req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:33.328802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05740a7d1d0c40bbbe3408de006c3d57', 'request_id': 'req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:33.343271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] The token cache is being invalidate because project 05740a7d1d0c40bbbe3408de006c3d57 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:33.343771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:33.344703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-77acbe30-d34a-4a79-a9a6-75a449b3eb1f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:33.358029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1221/2446] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:33 2022] DELETE /identity/v3/projects/05740a7d1d0c40bbbe3408de006c3d57 => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:33.370323 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac26708-1c43-42bc-b681-75c054cdb568 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.370915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac26708-1c43-42bc-b681-75c054cdb568 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.370915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ac26708-1c43-42bc-b681-75c054cdb568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.392595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7ac26708-1c43-42bc-b681-75c054cdb568 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:33.393763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7ac26708-1c43-42bc-b681-75c054cdb568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:33.466410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1226/2447] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:33.474505 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa2a63d-1a3e-4a5a-8e9f-ec45ae6bf441 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.474784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa2a63d-1a3e-4a5a-8e9f-ec45ae6bf441 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.475060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa2a63d-1a3e-4a5a-8e9f-ec45ae6bf441 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.505990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1aa2a63d-1a3e-4a5a-8e9f-ec45ae6bf441 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:33.507321 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1aa2a63d-1a3e-4a5a-8e9f-ec45ae6bf441 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:33.516144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1222/2448] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:33.520541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac26708-1c43-42bc-b681-75c054cdb568 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.530653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ac26708-1c43-42bc-b681-75c054cdb568 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:33.535952 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:33.536578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.536936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.537268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.538277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:33.538673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:33.540250 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:33.556884 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1227/2449] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:33.559058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa2a63d-1a3e-4a5a-8e9f-ec45ae6bf441 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.568426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa2a63d-1a3e-4a5a-8e9f-ec45ae6bf441 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:33.572989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:33.573596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.573867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.574172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.574798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:33.576701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:33.605724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1223/2450] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:33 2022] POST /identity/v3/projects => generated 381 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:33.610067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.620922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49caf5e-c705-4e6e-9de5-e978269eb2dc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:33.682790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:33.683299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.683567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.683857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.684484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:33.686431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:33.714243 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1228/2451] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:33 2022] POST /identity/v3/users => generated 291 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:33.715459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.726163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be76e1e-6354-41a4-b025-c085c2902ccd admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:33.731064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:33.731675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.731952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.732348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.733129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:33.734820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:33.745769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1224/2452] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:33.751139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.761397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86419c9e-78ba-4e70-b54f-bb0025602379 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:33.766045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f1075d-2adb-4edc-a3ad-c7dad8fe9797 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:33.766702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1075d-2adb-4edc-a3ad-c7dad8fe9797 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.767097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1075d-2adb-4edc-a3ad-c7dad8fe9797 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.767373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f1075d-2adb-4edc-a3ad-c7dad8fe9797 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.768022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f1075d-2adb-4edc-a3ad-c7dad8fe9797 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:33.769629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f1075d-2adb-4edc-a3ad-c7dad8fe9797 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:33.780921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1229/2453] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:33.785255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.795914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d01ede-59a0-475e-84d4-c317761142de admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:33.803531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:33.804419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.804696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.804962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] PATH_INFO: `/v3/projects/ed8465777e4a416e80829abcb2a21ba0/users/5ae5f689096b43ff89b4e223fd77d776/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.824025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ed8465777e4a416e80829abcb2a21ba0, user_id=5ae5f689096b43ff89b4e223fd77d776, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:33.827437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:33.842508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1225/2454] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:33 2022] PUT /identity/v3/projects/ed8465777e4a416e80829abcb2a21ba0/users/5ae5f689096b43ff89b4e223fd77d776/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:33.854329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69a9d8-c218-4b2d-bb10-3b6f94a985ff None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:33.854615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69a9d8-c218-4b2d-bb10-3b6f94a985ff None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:33.854913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69a9d8-c218-4b2d-bb10-3b6f94a985ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:33.886980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4e69a9d8-c218-4b2d-bb10-3b6f94a985ff None None] MFA Rules not processed for user `5ae5f689096b43ff89b4e223fd77d776`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:33.888226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4e69a9d8-c218-4b2d-bb10-3b6f94a985ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:33.927708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:33.962814 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1230/2455] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:33 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:33.968000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:33.997653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:34.040162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63bb4e2-b7f6-4d08-96eb-d202985547b4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.047294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.048106 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.048467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.048739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.049366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:34.051367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:34.084208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1226/2456] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:33 2022] POST /identity/v3/projects => generated 379 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:34.085589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69a9d8-c218-4b2d-bb10-3b6f94a985ff None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:34.097128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69a9d8-c218-4b2d-bb10-3b6f94a985ff None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.155616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.156257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.156531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.156798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.157479 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:34.159526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:34.184984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1231/2457] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:34 2022] POST /identity/v3/users => generated 296 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:34.189318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:34.200195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a910092-0d8a-4b26-88db-8d37d28d9170 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.205446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.205989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.206289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.207248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.207901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:34.209493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:34.220748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1227/2458] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:34.225077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:34.235855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8618fe-8e3f-4218-93ac-8adb6a935845 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.240856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.241517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.241793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.242099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] PATH_INFO: `/v3/projects/578cfe8d3fa64cf48a575bd565d25916/users/fa15606c9508446c8f41ca7c273bb7d1/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.258369 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] RBAC: Authorizing `identity:create_grant(project_id=578cfe8d3fa64cf48a575bd565d25916, user_id=fa15606c9508446c8f41ca7c273bb7d1, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:34.261034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:34.275221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1232/2459] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:34 2022] PUT /identity/v3/projects/578cfe8d3fa64cf48a575bd565d25916/users/fa15606c9508446c8f41ca7c273bb7d1/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:34.279358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:34.291461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-362e6c3b-2181-4b5e-8196-8f775c55e902 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.353916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b36bc6-8a6d-41c3-827d-cb029d07709c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.354555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b36bc6-8a6d-41c3-827d-cb029d07709c admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.354832 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b36bc6-8a6d-41c3-827d-cb029d07709c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.355101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b36bc6-8a6d-41c3-827d-cb029d07709c admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.355739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b36bc6-8a6d-41c3-827d-cb029d07709c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:34.357378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b36bc6-8a6d-41c3-827d-cb029d07709c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:34.367624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1228/2460] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:34 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:34.372105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:34.383149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c303cf-8377-48cf-99dc-27134cc50843 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.387836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.388457 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.388754 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.389026 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] PATH_INFO: `/v3/projects/578cfe8d3fa64cf48a575bd565d25916/users/fa15606c9508446c8f41ca7c273bb7d1/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.390948 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=578cfe8d3fa64cf48a575bd565d25916, user_id=fa15606c9508446c8f41ca7c273bb7d1, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:34.393770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:34.410117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1233/2461] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:34 2022] PUT /identity/v3/projects/578cfe8d3fa64cf48a575bd565d25916/users/fa15606c9508446c8f41ca7c273bb7d1/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:21:34.416160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8ae3c-ed66-4ea4-8d5f-32060483d8f0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.416446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8ae3c-ed66-4ea4-8d5f-32060483d8f0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.416748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be8ae3c-ed66-4ea4-8d5f-32060483d8f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.448370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-1be8ae3c-ed66-4ea4-8d5f-32060483d8f0 None None] MFA Rules not processed for user `fa15606c9508446c8f41ca7c273bb7d1`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:34.449602 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1be8ae3c-ed66-4ea4-8d5f-32060483d8f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:34.522704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1229/2462] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:34 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:34.536070 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:34.538170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:34.601437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13e498d7-a372-42b2-b40e-81e020db6bd6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.605969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.606624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.606896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.607319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:34.617432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:34.618975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:34.630193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1234/2463] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:34 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:34.983579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8ae3c-ed66-4ea4-8d5f-32060483d8f0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:34.993693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be8ae3c-ed66-4ea4-8d5f-32060483d8f0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:34.998358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa15606c9508446c8f41ca7c273bb7d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '578cfe8d3fa64cf48a575bd565d25916', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:34.998985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:34.999261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:34.999868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.000517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:35.000802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:35.002739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:35.014533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1230/2464] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:35.019162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:35.028788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae8219c-f2b7-45d8-8532-071868f8d354 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:35.033643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa15606c9508446c8f41ca7c273bb7d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '578cfe8d3fa64cf48a575bd565d25916', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:35.034404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.034692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.034962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.035589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:35.037592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:35.068595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1235/2465] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:35 2022] POST /identity/v3/projects => generated 355 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:35.072819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:35.083319 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b643c64-f2a5-4adb-8268-4d07d9f64890 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:35.088543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:35.150725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa15606c9508446c8f41ca7c273bb7d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '578cfe8d3fa64cf48a575bd565d25916', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:35.151489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.151772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.152048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.152701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:35.153007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:35.154658 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:35.167661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1231/2466] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:35.205075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:35.216064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e548dc7-b491-4e40-acee-f78eefeca639 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:35.220697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'fa15606c9508446c8f41ca7c273bb7d1', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '578cfe8d3fa64cf48a575bd565d25916', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:35.221326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.221601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.221871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] PATH_INFO: `/v3/projects/e058820bfff241a384b4ff7d1177bb3e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.223031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorizing `identity:delete_project(project_id=e058820bfff241a384b4ff7d1177bb3e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:35.224962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:35.227483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:35.280372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e058820bfff241a384b4ff7d1177bb3e', 'request_id': 'req-4c0719fb-5eed-4650-909a-4989ae1025f4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.289794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e058820bfff241a384b4ff7d1177bb3e', 'request_id': 'req-4c0719fb-5eed-4650-909a-4989ae1025f4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.298647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e058820bfff241a384b4ff7d1177bb3e', 'request_id': 'req-4c0719fb-5eed-4650-909a-4989ae1025f4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.310891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] The token cache is being invalidate because project e058820bfff241a384b4ff7d1177bb3e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:35.311274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.311939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4c0719fb-5eed-4650-909a-4989ae1025f4 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.314052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1236/2467] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:35 2022] DELETE /identity/v3/projects/e058820bfff241a384b4ff7d1177bb3e => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:35.521499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:35.523718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:35.593938 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c10ba6b-1f05-4356-b771-37d2db348a53 tempest-VolumeQuotaClassesTest-686830107 tempest-VolumeQuotaClassesTest-686830107-project-admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:35.651296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:35.652070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.652538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.652830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] PATH_INFO: `/v3/users/5ae5f689096b43ff89b4e223fd77d776` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.653983 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5ae5f689096b43ff89b4e223fd77d776)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:35.655965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:35.705203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5ae5f689096b43ff89b4e223fd77d776'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.707315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:35.713568 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5ae5f689096b43ff89b4e223fd77d776'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.714672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5ae5f689096b43ff89b4e223fd77d776'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.715512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5ae5f689096b43ff89b4e223fd77d776'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.723175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5ae5f689096b43ff89b4e223fd77d776'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:35.740219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1232/2468] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:35 2022] DELETE /identity/v3/users/5ae5f689096b43ff89b4e223fd77d776 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:35.744878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.745360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.746080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.766387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:35.767736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:35.827097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1237/2469] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:35.837153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:35.838662 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:35.899515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4354f170-9784-4183-af7e-095e79ab7583 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:35.904041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a51835b4-c985-4d78-997e-d39c01e2586c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:35.904651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51835b4-c985-4d78-997e-d39c01e2586c service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:35.904922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51835b4-c985-4d78-997e-d39c01e2586c service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:35.905182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a51835b4-c985-4d78-997e-d39c01e2586c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:35.915669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51835b4-c985-4d78-997e-d39c01e2586c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:35.917124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a51835b4-c985-4d78-997e-d39c01e2586c service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:35.926758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1233/2470] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:35 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:36.327715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:36.329343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:36.341709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea7669fc-431d-4d5b-9fd2-a6468cbc56b6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:36.348198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:36.348833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:36.349107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:36.349381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] PATH_INFO: `/v3/projects/ed8465777e4a416e80829abcb2a21ba0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:36.350577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ed8465777e4a416e80829abcb2a21ba0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:36.352551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:36.409251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ed8465777e4a416e80829abcb2a21ba0', 'request_id': 'req-902504d0-d461-4391-9556-f4f7670c923f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.423194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ed8465777e4a416e80829abcb2a21ba0', 'request_id': 'req-902504d0-d461-4391-9556-f4f7670c923f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.424381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ed8465777e4a416e80829abcb2a21ba0', 'request_id': 'req-902504d0-d461-4391-9556-f4f7670c923f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.424893 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:36.425254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:36.425558 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:36.436125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] The token cache is being invalidate because project ed8465777e4a416e80829abcb2a21ba0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:36.436411 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.437056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.438745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1238/2471] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:36 2022] DELETE /identity/v3/projects/ed8465777e4a416e80829abcb2a21ba0 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:36.442871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:36.444188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:36.444328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] MFA Rules not processed for user `480c3f60b5e14b5f93e5dc921694aee4`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:36.445476 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:36.463282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:36.500599 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1234/2472] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:36 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:36.511221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902504d0-d461-4391-9556-f4f7670c923f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:36.516573 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:36.517264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:36.517621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:36.517933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] PATH_INFO: `/v3/users/fa15606c9508446c8f41ca7c273bb7d1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:36.519132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fa15606c9508446c8f41ca7c273bb7d1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:36.521057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:36.577648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fa15606c9508446c8f41ca7c273bb7d1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.584609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fa15606c9508446c8f41ca7c273bb7d1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.585314 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fa15606c9508446c8f41ca7c273bb7d1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.585922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fa15606c9508446c8f41ca7c273bb7d1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.594534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fa15606c9508446c8f41ca7c273bb7d1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:36.612199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1239/2473] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:21:36 2022] DELETE /identity/v3/users/fa15606c9508446c8f41ca7c273bb7d1 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:21:36.816462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:36.818058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:36.873968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86206aa-2945-43ab-9a4e-ab5668486694 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:36.888165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:36.889766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:36.908941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:36.933507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f939cff-21c4-4fc6-895c-3f55eef8f9ad service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:36.934269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f939cff-21c4-4fc6-895c-3f55eef8f9ad service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:36.934607 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f939cff-21c4-4fc6-895c-3f55eef8f9ad service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:36.934935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f939cff-21c4-4fc6-895c-3f55eef8f9ad service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:36.936717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7f939cff-21c4-4fc6-895c-3f55eef8f9ad service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:36.946604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f939cff-21c4-4fc6-895c-3f55eef8f9ad service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:36.948127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f939cff-21c4-4fc6-895c-3f55eef8f9ad service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:36.956808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1235/2474] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:36 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 141 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:36.964331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b742bff3-617f-4ea4-87d5-fe9c4a70e467 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:36.968987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:36.969625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:36.969958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:36.970273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] PATH_INFO: `/v3/projects/578cfe8d3fa64cf48a575bd565d25916` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:36.971421 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=578cfe8d3fa64cf48a575bd565d25916)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:36.973372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.014859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:37.025049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '578cfe8d3fa64cf48a575bd565d25916', 'request_id': 'req-9ecae099-2933-4fae-9de3-fcbb838750a0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:37.033137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '578cfe8d3fa64cf48a575bd565d25916', 'request_id': 'req-9ecae099-2933-4fae-9de3-fcbb838750a0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:37.039904 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '578cfe8d3fa64cf48a575bd565d25916', 'request_id': 'req-9ecae099-2933-4fae-9de3-fcbb838750a0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:37.050618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] The token cache is being invalidate because project 578cfe8d3fa64cf48a575bd565d25916 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:21:37.050902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:37.051622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ecae099-2933-4fae-9de3-fcbb838750a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:21:37.053283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1240/2475] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:21:36 2022] DELETE /identity/v3/projects/578cfe8d3fa64cf48a575bd565d25916 => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:21:37.059797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ed0d2-4244-49a5-94f5-b9a28c585315 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.060071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ed0d2-4244-49a5-94f5-b9a28c585315 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.060411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1ed0d2-4244-49a5-94f5-b9a28c585315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.080119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7b1ed0d2-4244-49a5-94f5-b9a28c585315 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:37.081325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7b1ed0d2-4244-49a5-94f5-b9a28c585315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:37.140667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1236/2476] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:37.148634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0c2c70-9fd3-4358-81d0-49136046a80a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.148916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0c2c70-9fd3-4358-81d0-49136046a80a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.149184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0c2c70-9fd3-4358-81d0-49136046a80a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.169815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1b0c2c70-9fd3-4358-81d0-49136046a80a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:37.171069 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1b0c2c70-9fd3-4358-81d0-49136046a80a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:37.179587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1241/2477] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:37.183833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ed0d2-4244-49a5-94f5-b9a28c585315 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:37.194306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1ed0d2-4244-49a5-94f5-b9a28c585315 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:37.198838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:37.199444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.199697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.199962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.200606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:21:37.200902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:37.202535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.214473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1237/2478] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:21:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:21:37.219124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0c2c70-9fd3-4358-81d0-49136046a80a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:37.228964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0c2c70-9fd3-4358-81d0-49136046a80a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:37.235108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:37.235804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.236075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.236339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.236969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:37.238910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.271402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1242/2479] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:21:37 2022] POST /identity/v3/projects => generated 369 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:37.275605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:37.287248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16b4e9ad-5e08-43d6-9402-f2ca5d1a95b9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:37.357762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:37.357762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.357762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.357762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.358127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:37.358127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.379095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1238/2480] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:21:37 2022] POST /identity/v3/users => generated 285 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:21:37.380346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:37.390835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d9d08e-2850-4d44-b8ff-4b09d5ddd87d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:37.395819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:37.396419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.396695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.396961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.397599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:37.399235 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.409650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1243/2481] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:37 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:37.413751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:37.424523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92ba978-da64-4e0b-8455-261e497668d8 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:37.429290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60503489-344c-4771-b371-dd20412128bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:37.429903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60503489-344c-4771-b371-dd20412128bd admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.430399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60503489-344c-4771-b371-dd20412128bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.430679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60503489-344c-4771-b371-dd20412128bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.431328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60503489-344c-4771-b371-dd20412128bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:37.432935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60503489-344c-4771-b371-dd20412128bd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.446029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1239/2482] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:21:37 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:21:37.448755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:37.459015 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d8a0a5-578b-4c73-bf7d-d88f61144ce9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:37.463879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:37.464845 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.465525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.465836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] PATH_INFO: `/v3/projects/bcb29fc33d004cedb9f0c390adc60ea6/users/377010d742504ed08c7ee6ed99a89577/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.480880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] RBAC: Authorizing `identity:create_grant(project_id=bcb29fc33d004cedb9f0c390adc60ea6, user_id=377010d742504ed08c7ee6ed99a89577, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:37.484120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.497402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1244/2483] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:21:37 2022] PUT /identity/v3/projects/bcb29fc33d004cedb9f0c390adc60ea6/users/377010d742504ed08c7ee6ed99a89577/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:21:37.569191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.569701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.569978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.601825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] MFA Rules not processed for user `377010d742504ed08c7ee6ed99a89577`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:21:37.602890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:37.633035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:37.671772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1240/2484] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:21:37 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:21:37.683918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:37.685798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:37.746497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7c0ec1c-598c-4bd5-9489-7fd0911e20cc admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:37.751184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f786e89b-f0ba-4bc1-8c42-5a1932231576 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:37.751898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786e89b-f0ba-4bc1-8c42-5a1932231576 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:37.752174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786e89b-f0ba-4bc1-8c42-5a1932231576 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:37.752563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f786e89b-f0ba-4bc1-8c42-5a1932231576 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:37.761537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f786e89b-f0ba-4bc1-8c42-5a1932231576 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:37.763011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f786e89b-f0ba-4bc1-8c42-5a1932231576 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:37.772693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1245/2485] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:37 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:47.668462 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=14 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:47.768364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=64,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:21:48.605789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:21:48.607282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:48.678548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9af759b-dc50-40c5-82eb-0b890081ec16 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:21:48.684069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:21:48.684680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:21:48.684956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:21:48.685219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:21:48.687033 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:21:48.699302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:21:48.700909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:21:48.710867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1241/2486] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:21:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:21:58.706681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=32 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:14.189244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d124f-31e4-472c-9d5d-a38ad30f5de0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:14.189816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d124f-31e4-472c-9d5d-a38ad30f5de0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:14.189816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-033d124f-31e4-472c-9d5d-a38ad30f5de0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:14.210173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-033d124f-31e4-472c-9d5d-a38ad30f5de0 None None] MFA Rules not processed for user `a6111a16f95d4bc58f84d9f60f703d80`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:14.211393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-033d124f-31e4-472c-9d5d-a38ad30f5de0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:14.270550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1246/2487] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:22:14 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:14.273069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:14.343358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e293654c-f944-48ef-8f4f-60e3b9a641f1 service placement] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:14.349020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:14.349677 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:14.349952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:14.350271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] PATH_INFO: `/v3/users/377010d742504ed08c7ee6ed99a89577` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:14.351525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] RBAC: Authorizing `identity:delete_user(user_id=377010d742504ed08c7ee6ed99a89577)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:14.353617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:14.406974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '377010d742504ed08c7ee6ed99a89577'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:14.414300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '377010d742504ed08c7ee6ed99a89577'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:14.414987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '377010d742504ed08c7ee6ed99a89577'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:14.415594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '377010d742504ed08c7ee6ed99a89577'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:14.424357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '377010d742504ed08c7ee6ed99a89577'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:14.442440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1242/2488] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:22:14 2022] DELETE /identity/v3/users/377010d742504ed08c7ee6ed99a89577 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:14.446762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:14.447117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:14.447585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:14.467574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:14.468853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:14.531725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1247/2489] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:22:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:14.541792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:14.543711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:14.603107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1226dfec-2e4f-4876-916d-40761a088c52 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:14.606727 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a49e66-d098-448e-8f10-59bc18a50ace service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:14.607341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a49e66-d098-448e-8f10-59bc18a50ace service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:14.607729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a49e66-d098-448e-8f10-59bc18a50ace service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:14.608005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a49e66-d098-448e-8f10-59bc18a50ace service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:14.617057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a49e66-d098-448e-8f10-59bc18a50ace service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:14.618593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a49e66-d098-448e-8f10-59bc18a50ace service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:14.628693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1243/2490] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:14.911958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:14.913388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:14.926667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23fdb409-0cf8-47d6-8d79-ea99f1eab683 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:14.931259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:14.931895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:14.932174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:14.932448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] PATH_INFO: `/v3/projects/bcb29fc33d004cedb9f0c390adc60ea6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:14.933588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bcb29fc33d004cedb9f0c390adc60ea6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:14.935641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.006405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bcb29fc33d004cedb9f0c390adc60ea6', 'request_id': 'req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:15.006405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bcb29fc33d004cedb9f0c390adc60ea6', 'request_id': 'req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:15.010607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bcb29fc33d004cedb9f0c390adc60ea6', 'request_id': 'req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:15.021518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] The token cache is being invalidate because project bcb29fc33d004cedb9f0c390adc60ea6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:22:15.021755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:15.022493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5a043a41-06a5-4ca6-9e5d-08404cbe53e5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:22:15.024320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1248/2491] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:22:14 2022] DELETE /identity/v3/projects/bcb29fc33d004cedb9f0c390adc60ea6 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:22:15.029491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97217028-fdd7-4634-8836-815660d8ec85 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.029722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97217028-fdd7-4634-8836-815660d8ec85 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.030020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97217028-fdd7-4634-8836-815660d8ec85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.050587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-97217028-fdd7-4634-8836-815660d8ec85 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:15.051799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-97217028-fdd7-4634-8836-815660d8ec85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:15.069641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:15.114670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1244/2492] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:22:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:15.122487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81530777-58f1-4024-9043-08b16a24d819 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.122772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81530777-58f1-4024-9043-08b16a24d819 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.123220 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81530777-58f1-4024-9043-08b16a24d819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.146359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-81530777-58f1-4024-9043-08b16a24d819 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:15.147217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-81530777-58f1-4024-9043-08b16a24d819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:15.155825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1249/2493] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:22:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:15.160129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97217028-fdd7-4634-8836-815660d8ec85 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:15.169937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97217028-fdd7-4634-8836-815660d8ec85 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:15.175237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:15.175954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.176304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.176644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.177363 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:22:15.177734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:15.179495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.196210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1245/2494] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:22:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:22:15.197463 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81530777-58f1-4024-9043-08b16a24d819 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:15.207687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81530777-58f1-4024-9043-08b16a24d819 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:15.212649 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:15.213254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.213525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.213791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.214520 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:15.216395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.238861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:15.249938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1250/2495] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:22:15 2022] POST /identity/v3/projects => generated 363 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:15.251221 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:15.262228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2cc967d-ad36-49e6-93f7-6689105dba3c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:15.328226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:15.328925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.329204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.329643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.330700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:15.332825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.362614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1246/2496] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:22:15 2022] POST /identity/v3/users => generated 282 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:22:15.363933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:15.375749 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db6c072-29da-4a91-9ec5-6b4966629dc3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:15.380947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:15.381564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.381842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.382158 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.382810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:15.384485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.395383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1251/2497] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:15 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:22:15.399660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:15.410441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db5171b-e2f7-48c1-a43c-fe01bc4b590c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:15.415244 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e076d3-323e-4343-9ec7-a0a81ebae75d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:15.415853 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e076d3-323e-4343-9ec7-a0a81ebae75d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.416128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e076d3-323e-4343-9ec7-a0a81ebae75d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.416396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e076d3-323e-4343-9ec7-a0a81ebae75d admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.417065 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e076d3-323e-4343-9ec7-a0a81ebae75d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:15.418723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e076d3-323e-4343-9ec7-a0a81ebae75d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.430358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1247/2498] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:22:15 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:22:15.434998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:15.445897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-605e96c0-8541-43f6-8e01-59bf2e206645 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:15.450857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:15.451552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.451831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.452388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] PATH_INFO: `/v3/projects/1c76477ccff14bc59acb0c83f11d9136/users/13dbd76f772c43308879402397d30133/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.467553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] RBAC: Authorizing `identity:create_grant(project_id=1c76477ccff14bc59acb0c83f11d9136, user_id=13dbd76f772c43308879402397d30133, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:15.471018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.486296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1252/2499] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:22:15 2022] PUT /identity/v3/projects/1c76477ccff14bc59acb0c83f11d9136/users/13dbd76f772c43308879402397d30133/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:22:15.497990 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.498314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.498574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.531044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] MFA Rules not processed for user `13dbd76f772c43308879402397d30133`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:15.532637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:15.602372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1248/2500] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:22:15 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:15.614277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:15.616143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:15.683681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cc455ac-6912-4125-b763-6f83dfa80dac admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:15.688017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:15.688622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:15.688927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:15.689156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:15.698486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:15.700913 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:15.712854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1253/2501] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:15 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:22:25.599575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:25.710434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=41,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:47.015519 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:47.017206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:47.076961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfa0b7aa-9b9a-44ad-9c2b-2d25be329597 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:47.080709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95aeb8f3-26fc-47ab-b87b-139625889f7f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:47.081311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aeb8f3-26fc-47ab-b87b-139625889f7f service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:47.081583 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aeb8f3-26fc-47ab-b87b-139625889f7f service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:47.081846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95aeb8f3-26fc-47ab-b87b-139625889f7f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:47.083751 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-95aeb8f3-26fc-47ab-b87b-139625889f7f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:47.163635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aeb8f3-26fc-47ab-b87b-139625889f7f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:47.165096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95aeb8f3-26fc-47ab-b87b-139625889f7f service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:47.176302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1249/2502] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:47 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:48.370393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.370952 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.371472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.371838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.372009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:48.384848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83fe5c3-7fbf-41bc-b649-129221f808f9 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.389955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.390671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.390943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.391228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.393146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:48.405268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:22:48.406590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:48.407695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.409318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.421016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1254/2503] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:48 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:48.462328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1250/2504] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:22:48 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:22:48.518790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:48.520340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:48.592764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9141c5ef-9db9-4827-9bdd-ab66ff0aa8c6 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:48.598505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:48.599114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:48.599389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:48.599659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:48.608893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:48.610523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:48.619306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1255/2505] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:22:48 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:22:54.378201 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:22:54.379932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:54.397071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9051cc-2295-45cd-8a0f-924c9dd6499c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:22:54.403374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4557ddd-cc8f-4e4c-9f0e-ecee0215bc77 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:22:54.403988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4557ddd-cc8f-4e4c-9f0e-ecee0215bc77 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:22:54.404267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4557ddd-cc8f-4e4c-9f0e-ecee0215bc77 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:22:54.404593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4557ddd-cc8f-4e4c-9f0e-ecee0215bc77 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:22:54.406674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c4557ddd-cc8f-4e4c-9f0e-ecee0215bc77 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:22:54.448047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:22:54.499411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4557ddd-cc8f-4e4c-9f0e-ecee0215bc77 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:22:54.501196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4557ddd-cc8f-4e4c-9f0e-ecee0215bc77 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:22:54.521823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1251/2506] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:22:54 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:22:58.615429 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=38 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:03.188860 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:03.263380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a070d3-592f-4e10-8a24-f34387a09b25 service swift] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:03.268637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:03.269283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:03.269562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:03.269838 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] PATH_INFO: `/v3/users/13dbd76f772c43308879402397d30133` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:03.271005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] RBAC: Authorizing `identity:delete_user(user_id=13dbd76f772c43308879402397d30133)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:03.272980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:03.326834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13dbd76f772c43308879402397d30133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.334424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13dbd76f772c43308879402397d30133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.335097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13dbd76f772c43308879402397d30133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.335702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13dbd76f772c43308879402397d30133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.344835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13dbd76f772c43308879402397d30133'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.363699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1256/2507] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:23:03 2022] DELETE /identity/v3/users/13dbd76f772c43308879402397d30133 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:23:03.368580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:03.369099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:03.369099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:03.388752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:03.389945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:03.456541 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1252/2508] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:23:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:03.468672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:03.470836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:03.537133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cf68c0d-954f-4a31-96a2-2832f054487d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:03.541266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bda65de-c365-4bd4-87a8-e9413a215e82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:03.541883 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bda65de-c365-4bd4-87a8-e9413a215e82 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:03.546244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bda65de-c365-4bd4-87a8-e9413a215e82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:03.546550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bda65de-c365-4bd4-87a8-e9413a215e82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:03.557916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bda65de-c365-4bd4-87a8-e9413a215e82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:03.559379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bda65de-c365-4bd4-87a8-e9413a215e82 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:03.569686 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1257/2509] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:03.849025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:03.850645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:03.863343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7054f800-4068-4e1a-9c85-a3363727ccc6 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:03.868078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:03.868714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:03.868984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:03.869250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] PATH_INFO: `/v3/projects/1c76477ccff14bc59acb0c83f11d9136` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:03.870945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] RBAC: Authorizing `identity:delete_project(project_id=1c76477ccff14bc59acb0c83f11d9136)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:03.873078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:03.919460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1c76477ccff14bc59acb0c83f11d9136', 'request_id': 'req-de26a72b-1343-438a-993b-193a7d1a330d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.928634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1c76477ccff14bc59acb0c83f11d9136', 'request_id': 'req-de26a72b-1343-438a-993b-193a7d1a330d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.937278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1c76477ccff14bc59acb0c83f11d9136', 'request_id': 'req-de26a72b-1343-438a-993b-193a7d1a330d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.949119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] The token cache is being invalidate because project 1c76477ccff14bc59acb0c83f11d9136 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:23:03.949403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.950057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-de26a72b-1343-438a-993b-193a7d1a330d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:23:03.951589 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1253/2510] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:23:03 2022] DELETE /identity/v3/projects/1c76477ccff14bc59acb0c83f11d9136 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:23:06.515641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ff22ee-1c9a-43ef-af03-cf161ce6c6a2 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:06.516204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ff22ee-1c9a-43ef-af03-cf161ce6c6a2 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:06.516204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50ff22ee-1c9a-43ef-af03-cf161ce6c6a2 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:06.517502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1258/2511] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:23:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:23:06.524135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83a5ef-1946-4dc2-bdc9-dd96a9b9f18b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:06.524642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83a5ef-1946-4dc2-bdc9-dd96a9b9f18b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:06.524642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf83a5ef-1946-4dc2-bdc9-dd96a9b9f18b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:06.564347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cf83a5ef-1946-4dc2-bdc9-dd96a9b9f18b None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:06.565685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cf83a5ef-1946-4dc2-bdc9-dd96a9b9f18b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:06.579177 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:06.636536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1254/2512] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:23:06 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:06.648623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ff22ee-1c9a-43ef-af03-cf161ce6c6a2 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:06.650370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-50ff22ee-1c9a-43ef-af03-cf161ce6c6a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:06.681723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:06.736593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50ff22ee-1c9a-43ef-af03-cf161ce6c6a2 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:06.740829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2a8e9f6-ad0c-480c-a5a7-2c5fa0649381 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:06.741768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a8e9f6-ad0c-480c-a5a7-2c5fa0649381 service glance] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:06.742046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a8e9f6-ad0c-480c-a5a7-2c5fa0649381 service glance] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:06.742331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2a8e9f6-ad0c-480c-a5a7-2c5fa0649381 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:06.753557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a8e9f6-ad0c-480c-a5a7-2c5fa0649381 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:06.755037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2a8e9f6-ad0c-480c-a5a7-2c5fa0649381 service glance] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:06.766450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1259/2513] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:06 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:06.840477 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2330082d-a377-4df8-949f-7c9275356162 None None] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:06.840745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2330082d-a377-4df8-949f-7c9275356162 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:06.841053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2330082d-a377-4df8-949f-7c9275356162 None None] PATH_INFO: `/` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:06.842453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1255/2514] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:23:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:23:06.847558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd83b57-5d48-4526-85bf-413475ef6a1e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:06.847931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd83b57-5d48-4526-85bf-413475ef6a1e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:06.848211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd83b57-5d48-4526-85bf-413475ef6a1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:06.890272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3cd83b57-5d48-4526-85bf-413475ef6a1e None None] MFA Rules not processed for user `cc2f041fe7194a3da199517b4cdbd265`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:06.892054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3cd83b57-5d48-4526-85bf-413475ef6a1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:06.953171 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1260/2515] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:23:06 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:06.966094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2330082d-a377-4df8-949f-7c9275356162 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:06.968914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2330082d-a377-4df8-949f-7c9275356162 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:06.985145 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2330082d-a377-4df8-949f-7c9275356162 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:06.989963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebde742-015a-4dd6-8206-2cf6d8941925 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:06.990598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebde742-015a-4dd6-8206-2cf6d8941925 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:06.990873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebde742-015a-4dd6-8206-2cf6d8941925 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:06.991142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebde742-015a-4dd6-8206-2cf6d8941925 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:06.998834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebde742-015a-4dd6-8206-2cf6d8941925 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:07.000278 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebde742-015a-4dd6-8206-2cf6d8941925 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:07.008876 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1256/2516] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:06 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:07.093798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6a3cc1-5510-4d39-bc72-a5d0a82f9ab6 None None] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:07.094136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6a3cc1-5510-4d39-bc72-a5d0a82f9ab6 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:07.094432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b6a3cc1-5510-4d39-bc72-a5d0a82f9ab6 None None] PATH_INFO: `/` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:07.096396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1261/2517] 10.222.0.61 () {54 vars in 921 bytes} [Thu Sep 29 02:23:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 29 02:23:07.101898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:07.102231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:07.102503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:07.124264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] MFA Rules not processed for user `bc807d5f532b4e4c9953ba5e7eb878bb`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:07.125469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:07.137005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1257/2518] 10.222.0.61 () {58 vars in 1018 bytes} [Thu Sep 29 02:23:07 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:07.147895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6a3cc1-5510-4d39-bc72-a5d0a82f9ab6 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:07.160892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b6a3cc1-5510-4d39-bc72-a5d0a82f9ab6 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:07.165793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5449e62-3b06-46c8-9878-0991effe942e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:07.166417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5449e62-3b06-46c8-9878-0991effe942e service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:07.166681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5449e62-3b06-46c8-9878-0991effe942e service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:07.166969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5449e62-3b06-46c8-9878-0991effe942e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:07.178191 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5449e62-3b06-46c8-9878-0991effe942e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:07.179630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5449e62-3b06-46c8-9878-0991effe942e service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:07.191984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1262/2519] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:07 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:07.256979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:07.258491 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:07.331805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c636ea7-49d9-4f0f-acc1-a215819f61fd None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:07.337955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:07.338605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:07.338882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:07.339154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:07.341097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:07.352725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:07.354324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:07.364253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1258/2520] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:07 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:09.193751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4351b6-0c5f-46be-a883-147b3d2643da None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:09.194308 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4351b6-0c5f-46be-a883-147b3d2643da None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:09.194411 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4351b6-0c5f-46be-a883-147b3d2643da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:09.244464 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8e4351b6-0c5f-46be-a883-147b3d2643da None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:09.245670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8e4351b6-0c5f-46be-a883-147b3d2643da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:09.306834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=100 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:09.312054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1263/2521] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:23:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:09.324877 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:09.326668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:09.415032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:09.420176 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bc72fb-fa20-4006-bcbb-d3a7c2240c20 service placement] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:09.425523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97ed979-dc87-4011-a3d6-56742e838ea7 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:09.426215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ed979-dc87-4011-a3d6-56742e838ea7 service swift] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:09.426494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ed979-dc87-4011-a3d6-56742e838ea7 service swift] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:09.426761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97ed979-dc87-4011-a3d6-56742e838ea7 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:09.434815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ed979-dc87-4011-a3d6-56742e838ea7 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:09.436453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97ed979-dc87-4011-a3d6-56742e838ea7 service swift] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:09.444780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1259/2522] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:23:09 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:23:19.441543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:27.835695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4351b6-0c5f-46be-a883-147b3d2643da None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:27.846770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4351b6-0c5f-46be-a883-147b3d2643da None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:27.850684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:27.851296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:27.851621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:27.851886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:27.853914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:27.871507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:27.872959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:27.883352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1264/2523] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:27 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:29.561001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:29.561493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:29.561493 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:29.590208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:23:29.591431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:29.598948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1260/2524] 10.222.0.61 () {58 vars in 967 bytes} [Thu Sep 29 02:23:29 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:23:29.612016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:29.625539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09de3c96-b707-4a5e-916c-bbfa8c2b1b1b service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:29.632746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:29.633372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:29.633647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:29.633917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:29.644371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:29.646822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:29.657440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1265/2525] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:23:29 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:23:34.302371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:23:34.304522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:34.322381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ce9565-119d-4e57-a1f7-692454741dea None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:23:34.328508 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:23:34.329118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:23:34.329394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:23:34.329661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:23:34.331464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:23:34.419254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:23:34.421083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:23:34.440484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1261/2526] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:23:34 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:23:39.654065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=12 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:23:44.437705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=44 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:00.794385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:00.808537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f6a4b0d-d601-453d-9e91-68f18c8c46fb service swift] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:00.815451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:00.816338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:00.816945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:00.817272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:00.819507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:00.834048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:00.835781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:00.847396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1266/2527] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:00 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:01.207135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:01.209229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:01.224304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bb8db69-eabf-4d85-9b81-430fe499c528 service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:01.231773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:01.232392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:01.232669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:01.232987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:01.235059 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:01.321991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:01.323642 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:01.344262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1262/2528] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:01 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:06.832208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:06.842678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df96d78c-4628-4bc2-8d9b-ecec7a71523a service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:06.847032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:06.847627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:06.847902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:06.848168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:06.850170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:06.928240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:06.929678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:06.940743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1267/2529] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:10.410141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:10.411651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:10.426141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26223b9b-76a6-43a4-a6b2-a360f50d273e service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:10.430753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:10.431622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:10.432053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:10.432384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] PATH_INFO: `/v3/users/480c3f60b5e14b5f93e5dc921694aee4` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:10.433725 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=480c3f60b5e14b5f93e5dc921694aee4)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:10.435872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:10.485791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '480c3f60b5e14b5f93e5dc921694aee4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.492464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '480c3f60b5e14b5f93e5dc921694aee4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.493110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '480c3f60b5e14b5f93e5dc921694aee4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.493696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '480c3f60b5e14b5f93e5dc921694aee4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.503948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-577da652-6c3e-406b-afcd-96f6e273a0d7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '480c3f60b5e14b5f93e5dc921694aee4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.528634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1263/2530] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:10 2022] DELETE /identity/v3/users/480c3f60b5e14b5f93e5dc921694aee4 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:10.711118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:10.722145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d99042ce-7f13-428a-86f9-d4953e618996 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:10.787335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:10.787972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:10.788272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:10.788541 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] PATH_INFO: `/v3/projects/e72ef47313ef4a8582a9e060d5597fc0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:10.789658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e72ef47313ef4a8582a9e060d5597fc0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:10.791622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:10.842839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e72ef47313ef4a8582a9e060d5597fc0', 'request_id': 'req-45d4b78d-563c-41f2-9078-2869a9ed0a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.851259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e72ef47313ef4a8582a9e060d5597fc0', 'request_id': 'req-45d4b78d-563c-41f2-9078-2869a9ed0a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.859167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e72ef47313ef4a8582a9e060d5597fc0', 'request_id': 'req-45d4b78d-563c-41f2-9078-2869a9ed0a59'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.871688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] The token cache is being invalidate because project e72ef47313ef4a8582a9e060d5597fc0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:10.872007 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.872671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-45d4b78d-563c-41f2-9078-2869a9ed0a59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:10.874219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1268/2531] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:10 2022] DELETE /identity/v3/projects/e72ef47313ef4a8582a9e060d5597fc0 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:10.879196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537798ab-3128-445a-a3d0-9b9795014dbe None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:10.879481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537798ab-3128-445a-a3d0-9b9795014dbe None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:10.879749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-537798ab-3128-445a-a3d0-9b9795014dbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:10.899979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-537798ab-3128-445a-a3d0-9b9795014dbe None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:10.901173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-537798ab-3128-445a-a3d0-9b9795014dbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:10.941362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:10.967396 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1264/2532] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:10.976333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a03021-7c8c-44d4-8e87-433449ade550 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:10.976585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a03021-7c8c-44d4-8e87-433449ade550 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:10.976880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a03021-7c8c-44d4-8e87-433449ade550 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:10.997973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e6a03021-7c8c-44d4-8e87-433449ade550 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:10.999267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e6a03021-7c8c-44d4-8e87-433449ade550 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:11.007458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1269/2533] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:24:11.013126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537798ab-3128-445a-a3d0-9b9795014dbe None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.024919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-537798ab-3128-445a-a3d0-9b9795014dbe None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.029866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.030502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.030781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.031071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.031720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:11.032021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.033649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.045676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1265/2534] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:11.052333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a03021-7c8c-44d4-8e87-433449ade550 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.056377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:11.060592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a03021-7c8c-44d4-8e87-433449ade550 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.067543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.068144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.068768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.069102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.069759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.072263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.114523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1270/2535] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:11 2022] POST /identity/v3/projects => generated 403 bytes in 65 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:11.118752 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.129277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e71b220d-79d2-45cf-9ddf-41b408443501 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.199461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.199741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.200132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.200423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.201092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.204133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.231296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1266/2536] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:11 2022] POST /identity/v3/users => generated 302 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:11.235607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.246603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaf9e483-aa6e-49ce-acb9-c546b5c6330d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.251790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.252418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.252687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.252950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.253614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.255291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.266240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1271/2537] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:11.270364 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.281672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe8e8813-948f-41fb-b4ae-9f0d60e6f7e1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.287802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f77c7bb-3900-4a18-ba36-ef27c7774078 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.288403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f77c7bb-3900-4a18-ba36-ef27c7774078 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.288683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f77c7bb-3900-4a18-ba36-ef27c7774078 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.288944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f77c7bb-3900-4a18-ba36-ef27c7774078 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.289585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f77c7bb-3900-4a18-ba36-ef27c7774078 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.291235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f77c7bb-3900-4a18-ba36-ef27c7774078 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.302709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1267/2538] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:11.306995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.317275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-681903ad-d662-4937-bd8d-560b8eacb78c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.322855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.323580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.323853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.324208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] PATH_INFO: `/v3/projects/489edb9ab6874ae8b1f747c0afadff8d/users/89c164f94a8b4122a2968750f82df0b8/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.340236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] RBAC: Authorizing `identity:create_grant(project_id=489edb9ab6874ae8b1f747c0afadff8d, user_id=89c164f94a8b4122a2968750f82df0b8, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.342926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.358803 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1272/2539] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:11 2022] PUT /identity/v3/projects/489edb9ab6874ae8b1f747c0afadff8d/users/89c164f94a8b4122a2968750f82df0b8/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:11.370506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb8b686-6d2c-44c5-8d56-77fc50d8b949 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.370850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb8b686-6d2c-44c5-8d56-77fc50d8b949 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.371188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcb8b686-6d2c-44c5-8d56-77fc50d8b949 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.400800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-fcb8b686-6d2c-44c5-8d56-77fc50d8b949 None None] MFA Rules not processed for user `89c164f94a8b4122a2968750f82df0b8`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:11.402196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-fcb8b686-6d2c-44c5-8d56-77fc50d8b949 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:11.473155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1268/2540] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:11 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:11.479348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.546575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a725038b-f06d-489d-9435-4abc9badf726 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.551659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.552285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.552550 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.552853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.553513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.555459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.586227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1273/2541] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:11 2022] POST /identity/v3/projects => generated 403 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:11.591048 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb8b686-6d2c-44c5-8d56-77fc50d8b949 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.605873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcb8b686-6d2c-44c5-8d56-77fc50d8b949 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.645624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:11.672237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.672855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.673128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.673485 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.674171 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.676174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.704821 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1269/2542] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:11 2022] POST /identity/v3/users => generated 308 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:24:11.706372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.718337 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed1b566-92b7-4af3-8e23-fe8ecf4c8938 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.723088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.723692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.723995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.724266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.724906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.726603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.738085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1274/2543] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:11.742357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.752803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eddabb5-da01-4b99-94b1-4814b0bb514d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.758629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.759287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.759564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.759833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] PATH_INFO: `/v3/projects/0aefd55e79b04f19b88a229e9a80e074/users/1b6bba70450f477e84ea54cf0a9f8d6b/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.780290 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0aefd55e79b04f19b88a229e9a80e074, user_id=1b6bba70450f477e84ea54cf0a9f8d6b, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.783225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.799564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1270/2544] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:11 2022] PUT /identity/v3/projects/0aefd55e79b04f19b88a229e9a80e074/users/1b6bba70450f477e84ea54cf0a9f8d6b/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:11.803897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.815772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9cb4297-5a1d-4071-b97b-52196dc3703b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.880804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-947aad02-653e-47c6-a70d-d0cb54eeac92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.881418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947aad02-653e-47c6-a70d-d0cb54eeac92 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.881694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947aad02-653e-47c6-a70d-d0cb54eeac92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.881967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-947aad02-653e-47c6-a70d-d0cb54eeac92 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.882645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947aad02-653e-47c6-a70d-d0cb54eeac92 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.884252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-947aad02-653e-47c6-a70d-d0cb54eeac92 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.895257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1275/2545] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:11 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:24:11.899547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:11.910839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9540e2d9-72c6-436d-b99a-3cd232809161 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:11.915425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:11.916078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.916352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.916622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] PATH_INFO: `/v3/projects/0aefd55e79b04f19b88a229e9a80e074/users/1b6bba70450f477e84ea54cf0a9f8d6b/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.919223 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0aefd55e79b04f19b88a229e9a80e074, user_id=1b6bba70450f477e84ea54cf0a9f8d6b, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:11.921882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:11.934282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1271/2546] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:11 2022] PUT /identity/v3/projects/0aefd55e79b04f19b88a229e9a80e074/users/1b6bba70450f477e84ea54cf0a9f8d6b/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:11.944915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828cf9c7-e794-4fa9-8dcd-32252b4778a7 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:11.945199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828cf9c7-e794-4fa9-8dcd-32252b4778a7 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:11.945474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-828cf9c7-e794-4fa9-8dcd-32252b4778a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:11.966221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:11.976005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-828cf9c7-e794-4fa9-8dcd-32252b4778a7 None None] MFA Rules not processed for user `1b6bba70450f477e84ea54cf0a9f8d6b`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:11.977609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-828cf9c7-e794-4fa9-8dcd-32252b4778a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:12.051388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1276/2547] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:11 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:24:12.062672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:12.064696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:12.126613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b578d4-d083-495d-b7e1-982efa55c5d1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:12.130538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a59cf56-b150-49f5-978f-6a3ff6c3b1da service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:12.131150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a59cf56-b150-49f5-978f-6a3ff6c3b1da service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:12.131422 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a59cf56-b150-49f5-978f-6a3ff6c3b1da service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:12.131709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a59cf56-b150-49f5-978f-6a3ff6c3b1da service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:12.142204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a59cf56-b150-49f5-978f-6a3ff6c3b1da service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:12.143672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a59cf56-b150-49f5-978f-6a3ff6c3b1da service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:12.158372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1272/2548] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:12 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:12.306891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828cf9c7-e794-4fa9-8dcd-32252b4778a7 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:12.371455 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-828cf9c7-e794-4fa9-8dcd-32252b4778a7 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:12.376062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:12.376695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:12.376964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:12.377236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] PATH_INFO: `/v3/users/89c164f94a8b4122a2968750f82df0b8` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:12.378407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] RBAC: Authorizing `identity:delete_user(user_id=89c164f94a8b4122a2968750f82df0b8)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:12.380324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:12.430697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '89c164f94a8b4122a2968750f82df0b8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:12.437774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '89c164f94a8b4122a2968750f82df0b8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:12.438495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '89c164f94a8b4122a2968750f82df0b8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:12.439107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '89c164f94a8b4122a2968750f82df0b8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:12.448195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '89c164f94a8b4122a2968750f82df0b8'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:12.466465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1277/2549] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:12 2022] DELETE /identity/v3/users/89c164f94a8b4122a2968750f82df0b8 => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:12.471162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:12.471443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:12.471739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:12.490902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:12.492097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:12.558203 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1273/2550] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:12 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:12.568478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:12.570208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:12.585921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:12.631977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a4f9c85-83f5-4ac6-94f1-8e66ca0a3767 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:12.636405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:12.637012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:12.637286 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:12.637551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:12.646690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:12.648124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:12.659708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1278/2551] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:12.954716 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:12.956304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:12.970055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-692ae02a-d5f9-47d9-b87c-cda2d21a9465 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:12.974803 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:12.975439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:12.975714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:12.976142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] PATH_INFO: `/v3/projects/489edb9ab6874ae8b1f747c0afadff8d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:12.977261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=489edb9ab6874ae8b1f747c0afadff8d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:12.979222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:13.023561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:13.039685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '489edb9ab6874ae8b1f747c0afadff8d', 'request_id': 'req-685fd818-cbc3-4414-b9af-40ce9af35dc4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.049073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '489edb9ab6874ae8b1f747c0afadff8d', 'request_id': 'req-685fd818-cbc3-4414-b9af-40ce9af35dc4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.055530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '489edb9ab6874ae8b1f747c0afadff8d', 'request_id': 'req-685fd818-cbc3-4414-b9af-40ce9af35dc4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.065903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] The token cache is being invalidate because project 489edb9ab6874ae8b1f747c0afadff8d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:13.066162 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.066785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.068332 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1274/2552] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:12 2022] DELETE /identity/v3/projects/489edb9ab6874ae8b1f747c0afadff8d => generated 0 bytes in 114 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:13.073772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:13.088687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9af437-5bc3-49bc-b145-d837f2d35406 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:13.155531 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:13.156162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:13.156425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:13.156680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] PATH_INFO: `/v3/users/1b6bba70450f477e84ea54cf0a9f8d6b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:13.157949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1b6bba70450f477e84ea54cf0a9f8d6b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:13.159889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:13.208447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1b6bba70450f477e84ea54cf0a9f8d6b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.215504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1b6bba70450f477e84ea54cf0a9f8d6b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.216163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1b6bba70450f477e84ea54cf0a9f8d6b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.216759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1b6bba70450f477e84ea54cf0a9f8d6b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.226280 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2418ef7c-80f6-437b-b75a-7a8ca591af67 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1b6bba70450f477e84ea54cf0a9f8d6b'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.244853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1279/2553] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:13 2022] DELETE /identity/v3/users/1b6bba70450f477e84ea54cf0a9f8d6b => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:13.535490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:13.537094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:13.551810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-685fd818-cbc3-4414-b9af-40ce9af35dc4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:13.615779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:13.616426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:13.616704 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:13.616981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] PATH_INFO: `/v3/projects/0aefd55e79b04f19b88a229e9a80e074` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:13.618845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0aefd55e79b04f19b88a229e9a80e074)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:13.621478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:13.669700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0aefd55e79b04f19b88a229e9a80e074', 'request_id': 'req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.678897 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0aefd55e79b04f19b88a229e9a80e074', 'request_id': 'req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.685575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0aefd55e79b04f19b88a229e9a80e074', 'request_id': 'req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.695754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] The token cache is being invalidate because project 0aefd55e79b04f19b88a229e9a80e074 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:13.696041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.696688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-19ff50f1-8b2e-4da4-a048-f26985a3ce4a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:13.698208 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1275/2554] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:13 2022] DELETE /identity/v3/projects/0aefd55e79b04f19b88a229e9a80e074 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:13.703427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d2333-7e61-4fec-a100-e9ffc2105212 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:13.703709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d2333-7e61-4fec-a100-e9ffc2105212 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:13.703978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa8d2333-7e61-4fec-a100-e9ffc2105212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:13.724050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-fa8d2333-7e61-4fec-a100-e9ffc2105212 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:13.725284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fa8d2333-7e61-4fec-a100-e9ffc2105212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:13.757174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:13.788986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1280/2555] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:13.798504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354071eb-0b5e-4137-86ed-3d8200d1c711 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:13.798863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354071eb-0b5e-4137-86ed-3d8200d1c711 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:13.799204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354071eb-0b5e-4137-86ed-3d8200d1c711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:13.819862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-354071eb-0b5e-4137-86ed-3d8200d1c711 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:13.821127 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-354071eb-0b5e-4137-86ed-3d8200d1c711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:13.829474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1276/2556] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:13.833906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d2333-7e61-4fec-a100-e9ffc2105212 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:13.844197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa8d2333-7e61-4fec-a100-e9ffc2105212 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:13.849857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:13.850470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:13.850744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:13.851017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:13.851667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:13.851969 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:13.853687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:13.866087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1281/2557] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:13.870348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354071eb-0b5e-4137-86ed-3d8200d1c711 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:13.881503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354071eb-0b5e-4137-86ed-3d8200d1c711 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:13.887647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:13.888264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:13.888539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:13.888805 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:13.889437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:13.891397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:13.924072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1277/2558] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:13 2022] POST /identity/v3/projects => generated 381 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:13.928419 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:13.938732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80ff1df4-0f8c-4243-aeed-7156d4dca8a6 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:13.996717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:13.997330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:13.997609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:13.997881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:13.998546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.000503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.024747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1282/2559] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:13 2022] POST /identity/v3/users => generated 291 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:14.030909 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.046652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9533b4-2488-4386-99d1-6bcd94a09769 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.054130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.054986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.055352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.055706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.056604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.059107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.070526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1278/2560] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:14 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:14.075897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.092098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a79d736-171d-4488-9cb8-8c355094abba admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.098240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9123571-e29a-42b7-bd55-85bdc89256e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.098854 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9123571-e29a-42b7-bd55-85bdc89256e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.099132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9123571-e29a-42b7-bd55-85bdc89256e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.099402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9123571-e29a-42b7-bd55-85bdc89256e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.100096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9123571-e29a-42b7-bd55-85bdc89256e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.101667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9123571-e29a-42b7-bd55-85bdc89256e5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.114508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1283/2561] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:14 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:14.119639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.131248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33981cef-6ffa-4f58-86f8-a8015e359d99 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.137817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.138559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.138836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.139113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] PATH_INFO: `/v3/projects/bd99e2c6c4be4de3b8ed01f910e23eb2/users/a2cd7d2dc8134e1eabffcac8542ef62e/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.141595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:14.164506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] RBAC: Authorizing `identity:create_grant(project_id=bd99e2c6c4be4de3b8ed01f910e23eb2, user_id=a2cd7d2dc8134e1eabffcac8542ef62e, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.167207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.183078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1279/2562] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:14 2022] PUT /identity/v3/projects/bd99e2c6c4be4de3b8ed01f910e23eb2/users/a2cd7d2dc8134e1eabffcac8542ef62e/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:14.193905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56603c-add5-48fb-b2eb-5fd0c3d5c197 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.194246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56603c-add5-48fb-b2eb-5fd0c3d5c197 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.194518 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d56603c-add5-48fb-b2eb-5fd0c3d5c197 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.224029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1d56603c-add5-48fb-b2eb-5fd0c3d5c197 None None] MFA Rules not processed for user `a2cd7d2dc8134e1eabffcac8542ef62e`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:14.225233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1d56603c-add5-48fb-b2eb-5fd0c3d5c197 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:14.296085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1284/2563] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:14 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:14.302705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.375240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228f085c-ac44-4b01-8ab8-f30a78c0a47a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.380116 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.380721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.380991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.381261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.381882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.383917 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.414313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1280/2564] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:14 2022] POST /identity/v3/projects => generated 381 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:14.418784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56603c-add5-48fb-b2eb-5fd0c3d5c197 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.430121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d56603c-add5-48fb-b2eb-5fd0c3d5c197 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.463003 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:14.498118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.498769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.499045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.499318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.499955 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.501938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.529632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1285/2565] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:14 2022] POST /identity/v3/users => generated 297 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:14.530953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.541763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35b75e1-079b-4b8c-bfd3-5a682fefc7e9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.549498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.549668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.550711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.551011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.551657 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.553304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.564475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1281/2566] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:14 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:14.569710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.580678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b78e89-9c38-4078-a3d7-2c5587fa5231 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.585527 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.586261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.586524 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.586794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] PATH_INFO: `/v3/projects/d06f85d90feb454a98619ab5a6b81d0d/users/b48d322f7ef4468d88a68e4978f4faf4/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.601807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d06f85d90feb454a98619ab5a6b81d0d, user_id=b48d322f7ef4468d88a68e4978f4faf4, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.604486 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.621848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1286/2567] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:14 2022] PUT /identity/v3/projects/d06f85d90feb454a98619ab5a6b81d0d/users/b48d322f7ef4468d88a68e4978f4faf4/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:14.626197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.638202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39ab1391-4696-4cc1-a538-045ae17a0a46 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.703132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8afd48-d941-42d4-be2c-93a99609557c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.703882 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8afd48-d941-42d4-be2c-93a99609557c admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.704161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8afd48-d941-42d4-be2c-93a99609557c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.704432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8afd48-d941-42d4-be2c-93a99609557c admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.705072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8afd48-d941-42d4-be2c-93a99609557c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.707015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8afd48-d941-42d4-be2c-93a99609557c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.717579 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1282/2568] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:14 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:14.721840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.733101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623c5c64-0a00-469b-a464-e01d3c35e005 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.738712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.739805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.739805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.740105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] PATH_INFO: `/v3/projects/d06f85d90feb454a98619ab5a6b81d0d/users/b48d322f7ef4468d88a68e4978f4faf4/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.742170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d06f85d90feb454a98619ab5a6b81d0d, user_id=b48d322f7ef4468d88a68e4978f4faf4, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.745035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.756693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1287/2569] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:14 2022] PUT /identity/v3/projects/d06f85d90feb454a98619ab5a6b81d0d/users/b48d322f7ef4468d88a68e4978f4faf4/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:14.769052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0154189-e1a6-452e-ad5b-9bcaf622dad2 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.769337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0154189-e1a6-452e-ad5b-9bcaf622dad2 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.769611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0154189-e1a6-452e-ad5b-9bcaf622dad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.803902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a0154189-e1a6-452e-ad5b-9bcaf622dad2 None None] MFA Rules not processed for user `b48d322f7ef4468d88a68e4978f4faf4`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:14.805590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a0154189-e1a6-452e-ad5b-9bcaf622dad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:14.853633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:14.882626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1283/2570] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:14 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:14.894351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:14.896132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:14.955442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c822456-c257-4f7f-9cb7-62b197d9c4d2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:14.959194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-439bcbbb-c280-4e29-b8d2-6000e8e5aa09 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:14.959808 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439bcbbb-c280-4e29-b8d2-6000e8e5aa09 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:14.960086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439bcbbb-c280-4e29-b8d2-6000e8e5aa09 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:14.960357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-439bcbbb-c280-4e29-b8d2-6000e8e5aa09 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:14.970024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439bcbbb-c280-4e29-b8d2-6000e8e5aa09 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:14.971638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-439bcbbb-c280-4e29-b8d2-6000e8e5aa09 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:14.986284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1288/2571] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:14 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:15.464608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0154189-e1a6-452e-ad5b-9bcaf622dad2 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:15.561210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0154189-e1a6-452e-ad5b-9bcaf622dad2 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:15.567365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:15.568540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:15.568820 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:15.569172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] PATH_INFO: `/v3/users/a2cd7d2dc8134e1eabffcac8542ef62e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:15.571070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2cd7d2dc8134e1eabffcac8542ef62e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:15.573119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:15.620538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2cd7d2dc8134e1eabffcac8542ef62e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:15.627806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2cd7d2dc8134e1eabffcac8542ef62e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:15.628481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2cd7d2dc8134e1eabffcac8542ef62e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:15.629109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2cd7d2dc8134e1eabffcac8542ef62e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:15.637843 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2cd7d2dc8134e1eabffcac8542ef62e'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:15.655036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1284/2572] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:15 2022] DELETE /identity/v3/users/a2cd7d2dc8134e1eabffcac8542ef62e => generated 0 bytes in 191 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:15.659567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:15.660057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:15.660447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:15.682059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:15.683311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:15.743857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1289/2573] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:15.753908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:15.755446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:15.814342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc36569-f125-4f62-bc70-36856c521fda admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:15.818554 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:15.819170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:15.819420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:15.819693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:15.829610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:15.831242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:15.838656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:15.844881 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1285/2574] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:16.168999 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:16.170625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:16.190650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d48864c0-bda7-40ea-8b7f-fcddb6f7bbbd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:16.196964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:16.197820 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:16.198136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:16.198559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] PATH_INFO: `/v3/projects/bd99e2c6c4be4de3b8ed01f910e23eb2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:16.200038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bd99e2c6c4be4de3b8ed01f910e23eb2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:16.202153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:16.235114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:16.261653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bd99e2c6c4be4de3b8ed01f910e23eb2', 'request_id': 'req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.270310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bd99e2c6c4be4de3b8ed01f910e23eb2', 'request_id': 'req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.277160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bd99e2c6c4be4de3b8ed01f910e23eb2', 'request_id': 'req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.288149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] The token cache is being invalidate because project bd99e2c6c4be4de3b8ed01f910e23eb2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:16.288477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.289082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.290612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1290/2575] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:16 2022] DELETE /identity/v3/projects/bd99e2c6c4be4de3b8ed01f910e23eb2 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:16.294785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:16.304312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed5e2cb-7d65-42ee-9eff-60428cab7b34 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:16.370297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:16.371005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:16.371433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:16.371706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] PATH_INFO: `/v3/users/b48d322f7ef4468d88a68e4978f4faf4` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:16.372929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b48d322f7ef4468d88a68e4978f4faf4)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:16.374941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:16.425081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b48d322f7ef4468d88a68e4978f4faf4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.432446 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b48d322f7ef4468d88a68e4978f4faf4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.433193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b48d322f7ef4468d88a68e4978f4faf4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.433939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b48d322f7ef4468d88a68e4978f4faf4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.443854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c81194ce-96ba-4e74-b70e-379c639bfb35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b48d322f7ef4468d88a68e4978f4faf4'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.468964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1286/2576] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:16 2022] DELETE /identity/v3/users/b48d322f7ef4468d88a68e4978f4faf4 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:16.741647 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:16.743221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:16.756617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc4f1a8-f8e0-4d47-870e-7a0408977c26 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:16.821760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:16.822411 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:16.822708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:16.822978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] PATH_INFO: `/v3/projects/d06f85d90feb454a98619ab5a6b81d0d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:16.824370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d06f85d90feb454a98619ab5a6b81d0d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:16.826391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:16.875391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd06f85d90feb454a98619ab5a6b81d0d', 'request_id': 'req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.884139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd06f85d90feb454a98619ab5a6b81d0d', 'request_id': 'req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.891396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd06f85d90feb454a98619ab5a6b81d0d', 'request_id': 'req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.902283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] The token cache is being invalidate because project d06f85d90feb454a98619ab5a6b81d0d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:16.902611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.903330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7ebdf311-0b9b-4fc6-b99b-7a1c0ae44d65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:16.904915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1291/2577] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:16 2022] DELETE /identity/v3/projects/d06f85d90feb454a98619ab5a6b81d0d => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:16.909959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe516d7-5521-4be2-a254-97d468f5de08 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:16.910504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe516d7-5521-4be2-a254-97d468f5de08 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:16.910504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe516d7-5521-4be2-a254-97d468f5de08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:16.930046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9fe516d7-5521-4be2-a254-97d468f5de08 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:16.931220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9fe516d7-5521-4be2-a254-97d468f5de08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:16.995090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1287/2578] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:17.003410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c755efb-e0a4-4321-841e-cc32955151e9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.003786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c755efb-e0a4-4321-841e-cc32955151e9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.004139 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c755efb-e0a4-4321-841e-cc32955151e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.026174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7c755efb-e0a4-4321-841e-cc32955151e9 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:17.027613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7c755efb-e0a4-4321-841e-cc32955151e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:17.038856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1292/2579] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:17.043178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe516d7-5521-4be2-a254-97d468f5de08 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.053452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe516d7-5521-4be2-a254-97d468f5de08 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.059330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.059924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.060195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.060483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.061129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:17.061430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.063099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.079165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1288/2580] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:17.080318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c755efb-e0a4-4321-841e-cc32955151e9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.089424 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c755efb-e0a4-4321-841e-cc32955151e9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.094046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.094643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.094919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.095189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.095816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.097699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.128318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1293/2581] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:17 2022] POST /identity/v3/projects => generated 373 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:17.132291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.142785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ebd3b6-0db7-4d32-8801-f2eba415e2af admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.146819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:17.206101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.206743 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.207118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.207298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.207922 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.209914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.235275 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1289/2582] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:17 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:17.239472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.251878 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5801fd88-3b6e-4bd3-afd5-755eb5323930 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.256831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.257552 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.257902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.258554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.259269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.261281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.268065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=14,keystone:SELECT=83,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:17.274027 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1294/2583] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:17 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:17.284066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.294395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13413fd4-3b6c-456e-879b-12c7311a84ac admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.299043 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0340c44-d045-4766-b152-0213da8b4713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.299670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0340c44-d045-4766-b152-0213da8b4713 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.299945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0340c44-d045-4766-b152-0213da8b4713 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.300214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0340c44-d045-4766-b152-0213da8b4713 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.300900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0340c44-d045-4766-b152-0213da8b4713 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.302692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0340c44-d045-4766-b152-0213da8b4713 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.314587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1290/2584] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:17 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:17.318949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.329099 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e519e7f2-d1f7-466d-92f7-f164fb7bff63 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.334812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.335466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.335736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.336005 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] PATH_INFO: `/v3/projects/1b035b91c6164d3c804143d6e58b4717/users/33fa8e8407e141d59fad9f7115a1f7c6/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.350803 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1b035b91c6164d3c804143d6e58b4717, user_id=33fa8e8407e141d59fad9f7115a1f7c6, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.353433 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.369275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1295/2585] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:17 2022] PUT /identity/v3/projects/1b035b91c6164d3c804143d6e58b4717/users/33fa8e8407e141d59fad9f7115a1f7c6/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:17.380123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85bfeff-80c0-4fd4-899d-fea06c5e7a42 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.380400 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85bfeff-80c0-4fd4-899d-fea06c5e7a42 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.380665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d85bfeff-80c0-4fd4-899d-fea06c5e7a42 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.409408 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-d85bfeff-80c0-4fd4-899d-fea06c5e7a42 None None] MFA Rules not processed for user `33fa8e8407e141d59fad9f7115a1f7c6`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:17.410708 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d85bfeff-80c0-4fd4-899d-fea06c5e7a42 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:17.479668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1291/2586] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:17 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:17.489355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.491148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:17.561095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1efcb2-ae1a-4e53-9157-eb012b948a4f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.565576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b40543-33db-4c76-93cf-51620c995f64 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.566346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b40543-33db-4c76-93cf-51620c995f64 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.566622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b40543-33db-4c76-93cf-51620c995f64 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.566888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b40543-33db-4c76-93cf-51620c995f64 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.576203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b40543-33db-4c76-93cf-51620c995f64 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.578018 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3b40543-33db-4c76-93cf-51620c995f64 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.587148 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1296/2587] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:17 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:17.610159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85bfeff-80c0-4fd4-899d-fea06c5e7a42 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.681553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d85bfeff-80c0-4fd4-899d-fea06c5e7a42 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.686989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.687624 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.687901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.688167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] PATH_INFO: `/v3/users/33fa8e8407e141d59fad9f7115a1f7c6` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.689395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=33fa8e8407e141d59fad9f7115a1f7c6)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.691576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.709982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:17.743271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '33fa8e8407e141d59fad9f7115a1f7c6'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:17.750649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '33fa8e8407e141d59fad9f7115a1f7c6'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:17.751454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '33fa8e8407e141d59fad9f7115a1f7c6'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:17.752121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '33fa8e8407e141d59fad9f7115a1f7c6'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:17.761445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '33fa8e8407e141d59fad9f7115a1f7c6'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:17.782858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1292/2588] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:17 2022] DELETE /identity/v3/users/33fa8e8407e141d59fad9f7115a1f7c6 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:17.786565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.787032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.787130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.807902 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:17.809319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:17.867944 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1297/2589] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:24:17.877945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:17.879557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:17.939260 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1957c91-77e7-4748-999f-4001e3ff7dd6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:17.943149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-756d3917-d0d8-4687-9dea-8742f48211f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:17.943754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d3917-d0d8-4687-9dea-8742f48211f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:17.944022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d3917-d0d8-4687-9dea-8742f48211f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:17.944288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-756d3917-d0d8-4687-9dea-8742f48211f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:17.953594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756d3917-d0d8-4687-9dea-8742f48211f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:17.955064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-756d3917-d0d8-4687-9dea-8742f48211f0 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:17.964514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1293/2590] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:18.257759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.259303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:18.272296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d475ba-bf51-456c-8198-215b1e901132 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:18.277644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:18.278510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.278783 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.279111 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] PATH_INFO: `/v3/projects/1b035b91c6164d3c804143d6e58b4717` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.280283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1b035b91c6164d3c804143d6e58b4717)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:18.282273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:18.330133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1b035b91c6164d3c804143d6e58b4717', 'request_id': 'req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:18.340214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1b035b91c6164d3c804143d6e58b4717', 'request_id': 'req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:18.346985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1b035b91c6164d3c804143d6e58b4717', 'request_id': 'req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:18.357445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] The token cache is being invalidate because project 1b035b91c6164d3c804143d6e58b4717 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:18.357735 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:18.358469 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5cf1c4e2-fa97-4f1e-a364-13a0c2430e16 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:18.360112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1298/2591] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:18 2022] DELETE /identity/v3/projects/1b035b91c6164d3c804143d6e58b4717 => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:18.365153 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b061cc-1373-4913-93ff-143460a2278a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.365437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b061cc-1373-4913-93ff-143460a2278a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.365713 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3b061cc-1373-4913-93ff-143460a2278a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.387723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-b3b061cc-1373-4913-93ff-143460a2278a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:18.388946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b3b061cc-1373-4913-93ff-143460a2278a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:18.451313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1294/2592] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:18.461135 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255abe0d-79aa-4aef-97c8-dfbf9a4fe970 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.461482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255abe0d-79aa-4aef-97c8-dfbf9a4fe970 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.461855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-255abe0d-79aa-4aef-97c8-dfbf9a4fe970 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.472509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:18.487886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-255abe0d-79aa-4aef-97c8-dfbf9a4fe970 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:18.489103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-255abe0d-79aa-4aef-97c8-dfbf9a4fe970 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:18.497657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1299/2593] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:18.502142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b061cc-1373-4913-93ff-143460a2278a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.514130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3b061cc-1373-4913-93ff-143460a2278a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:18.521385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:18.522217 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.522545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.522900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.523849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:18.524242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:18.526702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:18.541039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1295/2594] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:18.545394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255abe0d-79aa-4aef-97c8-dfbf9a4fe970 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.555620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-255abe0d-79aa-4aef-97c8-dfbf9a4fe970 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:18.561760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:18.562398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.562661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.562923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.563553 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:18.565491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:18.597521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1300/2595] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:18 2022] POST /identity/v3/projects => generated 359 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:18.603379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.614432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddc9a26-8edf-46e6-b879-e88fcbe06f81 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:18.637543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:18.681141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:18.681749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.682028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.682514 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.683212 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:18.685374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:18.708850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1296/2596] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:18 2022] POST /identity/v3/users => generated 280 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:18.713096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.726066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d36ef77-29af-4573-bffb-5bfa1feead20 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:18.733181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:18.733993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.734344 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.734697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.735643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:18.738077 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:18.751224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1301/2597] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:18 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:18.755337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.766781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5930a8a4-bdf9-4717-b7ba-7018d9ce01fc admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:18.772078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdf47f23-a5c0-4e20-941f-14c50683435d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:18.772707 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf47f23-a5c0-4e20-941f-14c50683435d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.772985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf47f23-a5c0-4e20-941f-14c50683435d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.773239 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdf47f23-a5c0-4e20-941f-14c50683435d admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.773883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf47f23-a5c0-4e20-941f-14c50683435d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:18.775539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdf47f23-a5c0-4e20-941f-14c50683435d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:18.787070 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1297/2598] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:18 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:18.791346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.801965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b328640-b4ba-4a38-af7e-f94892042f67 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:18.806982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:18.807629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.807906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.808178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] PATH_INFO: `/v3/projects/3685168805d345adab0a0692abf1918a/users/4e14003c31dc4982a1f4119008eabf93/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.823501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3685168805d345adab0a0692abf1918a, user_id=4e14003c31dc4982a1f4119008eabf93, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:18.826297 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:18.843079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1302/2599] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:18 2022] PUT /identity/v3/projects/3685168805d345adab0a0692abf1918a/users/4e14003c31dc4982a1f4119008eabf93/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:18.853605 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2b5db-49c3-4ec8-9dcc-d61b69c56ffa None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:18.853885 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2b5db-49c3-4ec8-9dcc-d61b69c56ffa None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:18.854314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a2b5db-49c3-4ec8-9dcc-d61b69c56ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:18.889405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-68a2b5db-49c3-4ec8-9dcc-d61b69c56ffa None None] MFA Rules not processed for user `4e14003c31dc4982a1f4119008eabf93`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:18.890739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-68a2b5db-49c3-4ec8-9dcc-d61b69c56ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:18.956860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1298/2600] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:18 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:18.991338 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:18.993173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:19.053978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59fc4975-a557-436e-b3db-ed9acc8bb3c4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:19.057967 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ae8e5f-dc71-4eec-9b57-031accc04475 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:19.058618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae8e5f-dc71-4eec-9b57-031accc04475 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:19.058900 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae8e5f-dc71-4eec-9b57-031accc04475 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:19.059293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ae8e5f-dc71-4eec-9b57-031accc04475 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:19.068612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ae8e5f-dc71-4eec-9b57-031accc04475 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:19.070039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ae8e5f-dc71-4eec-9b57-031accc04475 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:19.079268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1303/2601] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:18 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:19.094898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2b5db-49c3-4ec8-9dcc-d61b69c56ffa None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:19.162039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68a2b5db-49c3-4ec8-9dcc-d61b69c56ffa None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:19.167152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:19.167793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:19.168060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:19.168329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] PATH_INFO: `/v3/users/4e14003c31dc4982a1f4119008eabf93` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:19.169737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e14003c31dc4982a1f4119008eabf93)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:19.171717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:19.218401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:19.231488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e14003c31dc4982a1f4119008eabf93'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.239258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e14003c31dc4982a1f4119008eabf93'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.239994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e14003c31dc4982a1f4119008eabf93'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.240620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e14003c31dc4982a1f4119008eabf93'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.251098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e14003c31dc4982a1f4119008eabf93'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.269747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1299/2602] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:19 2022] DELETE /identity/v3/users/4e14003c31dc4982a1f4119008eabf93 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:19.276781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:19.277169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:19.277537 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:19.298880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:19.300319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:19.367697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1304/2603] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:19.378886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:19.380561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:19.452654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e372977a-5f93-4994-9c35-cc2664818af2 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:19.456620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7cc9cb6-c486-45e5-8724-2a99b483a2ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:19.457226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc9cb6-c486-45e5-8724-2a99b483a2ed service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:19.457499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc9cb6-c486-45e5-8724-2a99b483a2ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:19.457765 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7cc9cb6-c486-45e5-8724-2a99b483a2ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:19.467979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc9cb6-c486-45e5-8724-2a99b483a2ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:19.469436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7cc9cb6-c486-45e5-8724-2a99b483a2ed service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:19.479515 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1300/2604] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:19.774935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:19.776473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:19.790225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68eeaab9-8cfd-466d-a1cd-39ae884de421 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:19.796385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:19.797024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:19.797295 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:19.797569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] PATH_INFO: `/v3/projects/3685168805d345adab0a0692abf1918a` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:19.798840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] RBAC: Authorizing `identity:delete_project(project_id=3685168805d345adab0a0692abf1918a)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:19.800742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:19.844543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:19.855194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3685168805d345adab0a0692abf1918a', 'request_id': 'req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.867697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3685168805d345adab0a0692abf1918a', 'request_id': 'req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.875331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3685168805d345adab0a0692abf1918a', 'request_id': 'req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.886766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] The token cache is being invalidate because project 3685168805d345adab0a0692abf1918a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:19.886991 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.887633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ea9e6d16-21a8-4beb-98fc-c127b2d6e8ac admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:19.889145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1305/2605] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:19 2022] DELETE /identity/v3/projects/3685168805d345adab0a0692abf1918a => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:19.894848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3fde7-2bd1-479a-beba-11c81d2ca264 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:19.895155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3fde7-2bd1-479a-beba-11c81d2ca264 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:19.895426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9f3fde7-2bd1-479a-beba-11c81d2ca264 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:19.915711 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a9f3fde7-2bd1-479a-beba-11c81d2ca264 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:19.916913 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a9f3fde7-2bd1-479a-beba-11c81d2ca264 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:19.988904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1301/2606] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:24:19.996744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e491700c-332b-44e9-83cb-55dd04928fc0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:19.997100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e491700c-332b-44e9-83cb-55dd04928fc0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:19.997417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e491700c-332b-44e9-83cb-55dd04928fc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.018240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e491700c-332b-44e9-83cb-55dd04928fc0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:20.019446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e491700c-332b-44e9-83cb-55dd04928fc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:20.028832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1306/2607] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:20.033076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3fde7-2bd1-479a-beba-11c81d2ca264 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:20.042740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9f3fde7-2bd1-479a-beba-11c81d2ca264 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:20.047736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:20.048381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.048652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.048937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.050723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:20.050723 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:20.051691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:20.067320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1302/2608] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:20.071761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e491700c-332b-44e9-83cb-55dd04928fc0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:20.081395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e491700c-332b-44e9-83cb-55dd04928fc0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:20.086489 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:20.087226 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.087504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.087776 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.088410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:20.090389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:20.120914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1307/2609] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:20 2022] POST /identity/v3/projects => generated 373 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:20.125401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:20.140047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-451b1252-5427-4586-b31e-502d2cc86176 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:20.185349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:20.207442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:20.207442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.207442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.207442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.207809 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:20.209344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:20.234535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1303/2610] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:20 2022] POST /identity/v3/users => generated 287 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:20.238709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:20.249494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6495f32-c92e-45b3-8f49-bb1932b947c3 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:20.254648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:20.255290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.255653 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.255925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.256592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:20.258599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:20.270032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1308/2611] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:20.273891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:20.285504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd0e51f5-6037-40ed-b2dd-826e816bcf57 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:20.290411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbed3a76-4d31-41c2-b701-f26a83344713 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:20.291022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbed3a76-4d31-41c2-b701-f26a83344713 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.291417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbed3a76-4d31-41c2-b701-f26a83344713 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.291745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbed3a76-4d31-41c2-b701-f26a83344713 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.292469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbed3a76-4d31-41c2-b701-f26a83344713 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:20.294447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbed3a76-4d31-41c2-b701-f26a83344713 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:20.305775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1304/2612] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:20 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:20.310375 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:20.320758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80e22587-43af-4364-b053-8d1bb5bfb9d4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:20.325829 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:20.326598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.326874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.327146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] PATH_INFO: `/v3/projects/04420159627d4beeb44bb038e633a96d/users/d83a16ba92834ab6a23a4b4634711178/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.342593 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] RBAC: Authorizing `identity:create_grant(project_id=04420159627d4beeb44bb038e633a96d, user_id=d83a16ba92834ab6a23a4b4634711178, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:20.345283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:20.360598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1309/2613] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:20 2022] PUT /identity/v3/projects/04420159627d4beeb44bb038e633a96d/users/d83a16ba92834ab6a23a4b4634711178/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:20.365730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6df9d9-9bec-4333-a466-6fdc087f6250 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.366018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6df9d9-9bec-4333-a466-6fdc087f6250 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.366289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab6df9d9-9bec-4333-a466-6fdc087f6250 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.385449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ab6df9d9-9bec-4333-a466-6fdc087f6250 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:20.387012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ab6df9d9-9bec-4333-a466-6fdc087f6250 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:20.459618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1305/2614] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:20.470717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:20.472572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:20.538519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22e68db5-4810-46d0-82ab-f55e3c37773d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:20.544203 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:20.544842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:20.545345 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:20.545682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:20.557098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:20.558844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:20.569545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1310/2615] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:21.219976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:21.220569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:21.220772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:21.250674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] MFA Rules not processed for user `d83a16ba92834ab6a23a4b4634711178`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:21.251890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:21.319854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1306/2616] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:21 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:21.331519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:21.333020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:21.411690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bcc2de9-7926-4b00-8202-8058d977c769 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:21.415408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2711de65-8710-424a-9024-6db2f491b065 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:21.416105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2711de65-8710-424a-9024-6db2f491b065 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:21.416449 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2711de65-8710-424a-9024-6db2f491b065 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:21.416802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2711de65-8710-424a-9024-6db2f491b065 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:21.425782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2711de65-8710-424a-9024-6db2f491b065 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:21.429663 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2711de65-8710-424a-9024-6db2f491b065 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:21.436393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1311/2617] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:21 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:21.441468 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:21.442966 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:21.455190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf70f03-a1b3-4d9d-94cf-7f69446dd87f None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:21.459034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:21.459637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:21.459912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:21.460181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:21.461888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:21.474667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:21.476302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:21.486470 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1307/2618] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:22.834403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:22.835221 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:22.835504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:22.850325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:DELETE=4,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:22.857961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] MFA Rules not processed for user `d83a16ba92834ab6a23a4b4634711178`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:22.859750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:22.878627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1312/2619] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:22 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:23.230680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:23.240018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6463b9d-4be3-4a64-b712-8ac8ecd83658 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:23.244603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:23.245245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:23.245489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:23.246046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:23.248067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:23.260181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:23.262115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:23.271695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1308/2620] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:23 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:25.034844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.036400 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:25.051605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae20e61c-9961-440e-8ad9-37d79ffe49db None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.057199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.057840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.058184 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.058460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] PATH_INFO: `/v3/users/a6111a16f95d4bc58f84d9f60f703d80` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.059748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a6111a16f95d4bc58f84d9f60f703d80)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.061905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.116298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a6111a16f95d4bc58f84d9f60f703d80'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.124247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a6111a16f95d4bc58f84d9f60f703d80'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.124940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a6111a16f95d4bc58f84d9f60f703d80'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.125551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a6111a16f95d4bc58f84d9f60f703d80'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.135781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-4ca7b66d-c07e-4f29-b662-d8b4eb1f7080 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a6111a16f95d4bc58f84d9f60f703d80'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.163875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1313/2621] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:25 2022] DELETE /identity/v3/users/a6111a16f95d4bc58f84d9f60f703d80 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:25.352606 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.364919 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a51f906-3f74-481a-af76-a3c76cf3e6cd service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.394098 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:25.437943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.438721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.439002 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.439420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] PATH_INFO: `/v3/projects/9006f0bb109c4c96a387ff5c314f236e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.440660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] RBAC: Authorizing `identity:delete_project(project_id=9006f0bb109c4c96a387ff5c314f236e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.442824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.495782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9006f0bb109c4c96a387ff5c314f236e', 'request_id': 'req-836f8628-5dbd-42e2-bedc-cbf029a35edc'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.507104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9006f0bb109c4c96a387ff5c314f236e', 'request_id': 'req-836f8628-5dbd-42e2-bedc-cbf029a35edc'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.516021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9006f0bb109c4c96a387ff5c314f236e', 'request_id': 'req-836f8628-5dbd-42e2-bedc-cbf029a35edc'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.527571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] The token cache is being invalidate because project 9006f0bb109c4c96a387ff5c314f236e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:25.527859 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.528572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-836f8628-5dbd-42e2-bedc-cbf029a35edc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:25.534808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1309/2622] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:25 2022] DELETE /identity/v3/projects/9006f0bb109c4c96a387ff5c314f236e => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:25.539938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77dd359-e0c1-4669-9129-1bd559fdb7dd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.540228 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77dd359-e0c1-4669-9129-1bd559fdb7dd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.540504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b77dd359-e0c1-4669-9129-1bd559fdb7dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.564189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b77dd359-e0c1-4669-9129-1bd559fdb7dd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:25.565540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b77dd359-e0c1-4669-9129-1bd559fdb7dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:25.625782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1314/2623] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:25.634502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7009fcfd-bc1d-425e-8572-c9706a27fe7b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.634775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7009fcfd-bc1d-425e-8572-c9706a27fe7b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.635042 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7009fcfd-bc1d-425e-8572-c9706a27fe7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.655447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7009fcfd-bc1d-425e-8572-c9706a27fe7b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:25.656676 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7009fcfd-bc1d-425e-8572-c9706a27fe7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:25.665473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1310/2624] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:25.669661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77dd359-e0c1-4669-9129-1bd559fdb7dd None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.680255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b77dd359-e0c1-4669-9129-1bd559fdb7dd None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.684899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.685534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.685898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.686183 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.686985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:25.687285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.688858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.704880 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1315/2625] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:25.706395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7009fcfd-bc1d-425e-8572-c9706a27fe7b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.715744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7009fcfd-bc1d-425e-8572-c9706a27fe7b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.721586 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.722365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.722640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.722907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.723550 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.725464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.755761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1311/2626] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:25 2022] POST /identity/v3/projects => generated 405 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:25.760128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.771039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2c5e60-605d-4642-8554-186f02fda10b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.829959 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.830671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.830989 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.831288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.831940 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.833891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.842818 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:25.860439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1316/2627] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:25 2022] POST /identity/v3/users => generated 303 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:25.864615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.876125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4729992-86e4-4b1a-9b2a-1282ca353646 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.881266 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.881889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.882200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.882474 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.883122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.884801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.895808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1312/2628] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:25.899979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.910377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a25bbd-6fb9-48e7-82d9-9c3a5f26c38a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.915062 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282fed76-c888-4b2c-9735-74b382660200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.915664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fed76-c888-4b2c-9735-74b382660200 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.915937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fed76-c888-4b2c-9735-74b382660200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.916205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282fed76-c888-4b2c-9735-74b382660200 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.916846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282fed76-c888-4b2c-9735-74b382660200 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.918465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282fed76-c888-4b2c-9735-74b382660200 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.930133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1317/2629] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:25.934392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:25.944910 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-800ce285-21ec-425b-ab39-7631fd178d88 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:25.950067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:25.950798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.951081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.951353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] PATH_INFO: `/v3/projects/f5c0a5c311d24c14b19a64f93874a808/users/2f27f239a53843908670a2529d4f11eb/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:25.967504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] RBAC: Authorizing `identity:create_grant(project_id=f5c0a5c311d24c14b19a64f93874a808, user_id=2f27f239a53843908670a2529d4f11eb, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:25.970253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:25.984220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1313/2630] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:25 2022] PUT /identity/v3/projects/f5c0a5c311d24c14b19a64f93874a808/users/2f27f239a53843908670a2529d4f11eb/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:25.989642 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ec2d3-6ca3-48a9-a537-0a36aa6d1c0e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:25.989921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ec2d3-6ca3-48a9-a537-0a36aa6d1c0e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:25.990229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277ec2d3-6ca3-48a9-a537-0a36aa6d1c0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:26.010155 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-277ec2d3-6ca3-48a9-a537-0a36aa6d1c0e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:26.011719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-277ec2d3-6ca3-48a9-a537-0a36aa6d1c0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:26.072055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1318/2631] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:26.082600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:26.084621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:26.144228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dce2a57-be64-44c7-84cd-247f35248b8c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:26.148035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:26.148645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:26.148914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:26.149175 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:26.155469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:26.160846 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:26.162667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:26.173697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1314/2632] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:26 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:26.915793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ce502b-2393-422d-8f5a-de5bb4e4eb41 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:26.916071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ce502b-2393-422d-8f5a-de5bb4e4eb41 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:26.916339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ce502b-2393-422d-8f5a-de5bb4e4eb41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:26.944718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-62ce502b-2393-422d-8f5a-de5bb4e4eb41 None None] MFA Rules not processed for user `2f27f239a53843908670a2529d4f11eb`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:26.946039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-62ce502b-2393-422d-8f5a-de5bb4e4eb41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:27.023685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:27.024329 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1319/2633] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:26 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:27.033838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d8088c8-b282-4a90-beeb-8b8ca50e5190 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:27.038576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:27.039280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:27.039625 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:27.039907 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:27.040611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:27.042651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:27.077406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1315/2634] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:27 2022] POST /identity/v3/projects => generated 405 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:24:27.078693 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ce502b-2393-422d-8f5a-de5bb4e4eb41 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:27.096886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ce502b-2393-422d-8f5a-de5bb4e4eb41 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:27.161367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:27.161970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:27.162291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:27.162554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:27.163180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:27.165110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:27.183388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:27.195118 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1320/2635] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:27 2022] POST /identity/v3/users => generated 309 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:27.196953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:27.208636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b265ebec-e591-42dd-9d23-e9508ff8b4b6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:27.213914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:27.214569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:27.214849 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:27.215123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:27.215773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:27.217389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:27.228351 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1316/2636] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:27.233247 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:27.244045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb0e2f3-9598-434f-bc6d-0fef1adc66f8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:27.249130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:27.249781 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:27.250095 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:27.250382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] PATH_INFO: `/v3/projects/1ecba49ddd184892b59192b1fadc2b35/users/f7ba7425f9bd43a2a1d47a562f86a4ca/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:27.265712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ecba49ddd184892b59192b1fadc2b35, user_id=f7ba7425f9bd43a2a1d47a562f86a4ca, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:27.268456 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:27.283088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1321/2637] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:27 2022] PUT /identity/v3/projects/1ecba49ddd184892b59192b1fadc2b35/users/f7ba7425f9bd43a2a1d47a562f86a4ca/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:27.287128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:27.297714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7e3abb8-f64f-47f3-a0e3-1029ebd0a306 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:27.354699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed9da5b-1012-4db6-a84d-0465141688de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:27.355309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed9da5b-1012-4db6-a84d-0465141688de admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:27.355580 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed9da5b-1012-4db6-a84d-0465141688de admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:27.355847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed9da5b-1012-4db6-a84d-0465141688de admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:27.356480 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed9da5b-1012-4db6-a84d-0465141688de admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:27.358404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed9da5b-1012-4db6-a84d-0465141688de admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:27.368731 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1317/2638] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:27 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:27.373294 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:27.384024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35b9d34e-565d-4050-8f68-ca73d9d87b05 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:27.389254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:27.389899 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:27.390202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:27.390475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] PATH_INFO: `/v3/projects/1ecba49ddd184892b59192b1fadc2b35/users/f7ba7425f9bd43a2a1d47a562f86a4ca/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:27.392398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ecba49ddd184892b59192b1fadc2b35, user_id=f7ba7425f9bd43a2a1d47a562f86a4ca, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:27.395049 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:27.405835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1322/2639] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:27 2022] PUT /identity/v3/projects/1ecba49ddd184892b59192b1fadc2b35/users/f7ba7425f9bd43a2a1d47a562f86a4ca/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:28.456964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5319c-d735-469c-b596-9d3053f81b91 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:28.457582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5319c-d735-469c-b596-9d3053f81b91 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:28.457582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25a5319c-d735-469c-b596-9d3053f81b91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:28.496401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-25a5319c-d735-469c-b596-9d3053f81b91 None None] MFA Rules not processed for user `f7ba7425f9bd43a2a1d47a562f86a4ca`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:28.497900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-25a5319c-d735-469c-b596-9d3053f81b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:28.594690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1318/2640] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:28 2022] POST /identity/v3/auth/tokens => generated 3349 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:29.224174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:29.226110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:29.313933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6faf96f-6da8-4562-80db-8566e1a61497 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:29.321614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:29.322834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:29.322834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:29.323200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:29.325702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:29.340052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:29.342174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:29.354299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1323/2641] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:29 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:29.853721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5319c-d735-469c-b596-9d3053f81b91 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:29.901971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:29.919984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25a5319c-d735-469c-b596-9d3053f81b91 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:29.924791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:29.925390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:29.925660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:29.925925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] PATH_INFO: `/v3/users/2f27f239a53843908670a2529d4f11eb` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:29.927113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f27f239a53843908670a2529d4f11eb)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:29.929056 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:29.979862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f27f239a53843908670a2529d4f11eb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:29.987005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f27f239a53843908670a2529d4f11eb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:29.987666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f27f239a53843908670a2529d4f11eb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:29.988250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f27f239a53843908670a2529d4f11eb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:29.996313 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f27f239a53843908670a2529d4f11eb'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.014061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1319/2642] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:29 2022] DELETE /identity/v3/users/2f27f239a53843908670a2529d4f11eb => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:30.197679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:30.208512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f29828-e690-446a-a0c5-28931c21df3d service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:30.267906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:30.268587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:30.268865 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:30.269140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] PATH_INFO: `/v3/projects/f5c0a5c311d24c14b19a64f93874a808` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:30.270326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f5c0a5c311d24c14b19a64f93874a808)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:30.272255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:30.329594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f5c0a5c311d24c14b19a64f93874a808', 'request_id': 'req-b9a95a45-1e1e-4797-b20d-473e4780ee51'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.333170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:30.343151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f5c0a5c311d24c14b19a64f93874a808', 'request_id': 'req-b9a95a45-1e1e-4797-b20d-473e4780ee51'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.352453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f5c0a5c311d24c14b19a64f93874a808', 'request_id': 'req-b9a95a45-1e1e-4797-b20d-473e4780ee51'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.367407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] The token cache is being invalidate because project f5c0a5c311d24c14b19a64f93874a808 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:30.367824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.368588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.370495 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1324/2643] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:30 2022] DELETE /identity/v3/projects/f5c0a5c311d24c14b19a64f93874a808 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:30.374391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:30.376167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:30.389237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ad99d9-2ac9-42f5-81fe-e4e4c5fbf266 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:30.449739 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:30.450391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:30.450636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:30.450890 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] PATH_INFO: `/v3/users/f7ba7425f9bd43a2a1d47a562f86a4ca` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:30.452028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7ba7425f9bd43a2a1d47a562f86a4ca)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:30.453969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:30.503007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7ba7425f9bd43a2a1d47a562f86a4ca'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.509512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7ba7425f9bd43a2a1d47a562f86a4ca'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.510185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7ba7425f9bd43a2a1d47a562f86a4ca'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.510764 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7ba7425f9bd43a2a1d47a562f86a4ca'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.520019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-4a7f4a07-6c00-42c5-bce3-678a2f204dc6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7ba7425f9bd43a2a1d47a562f86a4ca'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.521082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=80,keystone:DELETE=18,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:30.547258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1320/2644] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:30 2022] DELETE /identity/v3/users/f7ba7425f9bd43a2a1d47a562f86a4ca => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:30.808480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:30.810327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:30.831309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a95a45-1e1e-4797-b20d-473e4780ee51 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:30.891107 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:30.891794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:30.892031 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:30.892298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] PATH_INFO: `/v3/projects/1ecba49ddd184892b59192b1fadc2b35` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:30.893402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ecba49ddd184892b59192b1fadc2b35)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:30.895336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:30.944590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ecba49ddd184892b59192b1fadc2b35', 'request_id': 'req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.953343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ecba49ddd184892b59192b1fadc2b35', 'request_id': 'req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.960261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ecba49ddd184892b59192b1fadc2b35', 'request_id': 'req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.971807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] The token cache is being invalidate because project 1ecba49ddd184892b59192b1fadc2b35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:30.972097 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.972761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-dd833b23-7fc4-46b3-b4eb-8ad83830d50a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:30.974367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1325/2645] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:30 2022] DELETE /identity/v3/projects/1ecba49ddd184892b59192b1fadc2b35 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:30.979417 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f00da-c4d1-4c38-adbe-fc1ab45f31b9 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:30.980448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f00da-c4d1-4c38-adbe-fc1ab45f31b9 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:30.980811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8f00da-c4d1-4c38-adbe-fc1ab45f31b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.001819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7a8f00da-c4d1-4c38-adbe-fc1ab45f31b9 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:31.003039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7a8f00da-c4d1-4c38-adbe-fc1ab45f31b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:31.072575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1321/2646] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:31.080549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bb76b1-eb72-4f18-ab65-76444b304a51 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.080830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bb76b1-eb72-4f18-ab65-76444b304a51 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.081104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4bb76b1-eb72-4f18-ab65-76444b304a51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.102204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-b4bb76b1-eb72-4f18-ab65-76444b304a51 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:31.103410 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b4bb76b1-eb72-4f18-ab65-76444b304a51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:31.111836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1326/2647] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:31.116342 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8f00da-c4d1-4c38-adbe-fc1ab45f31b9 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.127654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8f00da-c4d1-4c38-adbe-fc1ab45f31b9 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.132931 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.133582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.133878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.134213 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.134989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:31.135306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.137233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.151767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1322/2648] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:31.155982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bb76b1-eb72-4f18-ab65-76444b304a51 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.167343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4bb76b1-eb72-4f18-ab65-76444b304a51 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.172535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.173145 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.173417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.173685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.174364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.176282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.206979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1327/2649] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:31 2022] POST /identity/v3/projects => generated 367 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:31.211487 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.229262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32f11092-6e24-4145-a066-cc27b960b028 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.289507 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.290229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.290582 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.290940 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.291649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.294112 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.325523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1323/2650] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:31 2022] POST /identity/v3/users => generated 284 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:31.326815 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.337804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a249e1cf-e9ee-4a88-a0cc-a21b66ceb58a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.344255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.344945 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.345355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.345547 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.346172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.347819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.359208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1328/2651] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:24:31.363506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.374613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2017f0f-4cc8-487a-849b-2a6935d01b66 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.380009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a976a138-831f-4f63-8d00-c5323a80c10c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.380616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976a138-831f-4f63-8d00-c5323a80c10c admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.380886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976a138-831f-4f63-8d00-c5323a80c10c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.381149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976a138-831f-4f63-8d00-c5323a80c10c admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.381788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a976a138-831f-4f63-8d00-c5323a80c10c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.383444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a976a138-831f-4f63-8d00-c5323a80c10c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.396301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1324/2652] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:31 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:24:31.400460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.411110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f86a459b-d802-4393-a486-2d73dbf94840 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.416810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.417472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.417744 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.418050 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] PATH_INFO: `/v3/projects/275ae501c31a43d6bb7795288d4df76d/users/34658af7c8074a6fb9de14102e84f0b9/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.436292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=275ae501c31a43d6bb7795288d4df76d, user_id=34658af7c8074a6fb9de14102e84f0b9, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.439024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.472379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1329/2653] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:31 2022] PUT /identity/v3/projects/275ae501c31a43d6bb7795288d4df76d/users/34658af7c8074a6fb9de14102e84f0b9/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:31.476695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9c7db9-a4c9-494c-bff6-4e1d50fccdfb None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.476873 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9c7db9-a4c9-494c-bff6-4e1d50fccdfb None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.477101 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee9c7db9-a4c9-494c-bff6-4e1d50fccdfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.505845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ee9c7db9-a4c9-494c-bff6-4e1d50fccdfb None None] MFA Rules not processed for user `34658af7c8074a6fb9de14102e84f0b9`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:31.507097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ee9c7db9-a4c9-494c-bff6-4e1d50fccdfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:31.550357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:31.585053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1325/2654] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:31 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:31.589538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.600551 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=10,keystone:SELECT=87,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:31.666217 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea0f198-0458-40b8-b13b-1181de660da2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.672206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.672842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.673119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.673388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.674650 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.676281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.736980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1330/2655] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:31 2022] POST /identity/v3/projects => generated 369 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:24:31.741587 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9c7db9-a4c9-494c-bff6-4e1d50fccdfb None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.755817 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee9c7db9-a4c9-494c-bff6-4e1d50fccdfb None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.822317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.823402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.823402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.823594 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.824140 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.826196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.856929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1326/2656] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:31 2022] POST /identity/v3/users => generated 291 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:31.859237 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.870823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fadff7c7-5ee6-4adf-b28a-6f025574dab4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.876923 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.877557 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.877837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.878288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.879086 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.883002 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.903120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1331/2657] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:31 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:31.907982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:31.921848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b972043e-6ef8-455f-8e0c-dca9ea214413 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:31.930371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:31.931291 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:31.931572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:31.932086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] PATH_INFO: `/v3/projects/6ff5c212574e450087cd2cb774d07d61/users/1407e2ea43fc494896e317c5e8f64797/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:31.956762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ff5c212574e450087cd2cb774d07d61, user_id=1407e2ea43fc494896e317c5e8f64797, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:31.960088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:31.980636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1327/2658] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:31 2022] PUT /identity/v3/projects/6ff5c212574e450087cd2cb774d07d61/users/1407e2ea43fc494896e317c5e8f64797/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:31.984453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:32.003324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa78b4d7-ac00-4963-a046-f46c3b49e2a4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:32.080100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1305377-cc05-42a2-a167-f9c8da680f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:32.080825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1305377-cc05-42a2-a167-f9c8da680f4c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:32.081166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1305377-cc05-42a2-a167-f9c8da680f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:32.081439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1305377-cc05-42a2-a167-f9c8da680f4c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:32.082151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1305377-cc05-42a2-a167-f9c8da680f4c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:32.083747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1305377-cc05-42a2-a167-f9c8da680f4c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:32.094124 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1332/2659] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:31 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:24:32.098509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:32.109174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb8a8532-b81f-40d9-88d9-1c8af45946b2 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:32.113754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:32.114530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:32.114867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:32.115205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] PATH_INFO: `/v3/projects/6ff5c212574e450087cd2cb774d07d61/users/1407e2ea43fc494896e317c5e8f64797/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:32.117211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ff5c212574e450087cd2cb774d07d61, user_id=1407e2ea43fc494896e317c5e8f64797, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:32.119943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:32.130709 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1328/2660] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:32 2022] PUT /identity/v3/projects/6ff5c212574e450087cd2cb774d07d61/users/1407e2ea43fc494896e317c5e8f64797/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:32.141507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:32.141792 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:32.142140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:32.171931 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] MFA Rules not processed for user `1407e2ea43fc494896e317c5e8f64797`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:32.173334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:32.244875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1333/2661] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:32 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:32.255955 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:32.257650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:32.320200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d520c23-768d-4209-834e-0bd774dc9347 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:32.324601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:32.325222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:32.325496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:32.326053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:32.337182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:32.338830 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:32.349572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1329/2662] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:32 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:24:33.778357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:33.780269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:33.787000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:33.870745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5805a088-ce55-47e0-817d-e4d7cb894181 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:33.876598 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b16c3977-c35a-4bfe-8123-f1f3521918a8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:33.877214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c3977-c35a-4bfe-8123-f1f3521918a8 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:33.877637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c3977-c35a-4bfe-8123-f1f3521918a8 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:33.877763 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b16c3977-c35a-4bfe-8123-f1f3521918a8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:33.879657 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b16c3977-c35a-4bfe-8123-f1f3521918a8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:33.892306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c3977-c35a-4bfe-8123-f1f3521918a8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:33.893910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b16c3977-c35a-4bfe-8123-f1f3521918a8 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:33.905621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1334/2663] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:33 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:34.010324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:34.177385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:34.210045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf5d853-d552-4466-9e79-d64df38fe6cc service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:34.214338 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:34.214967 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:34.215237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:34.215498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] PATH_INFO: `/v3/users/34658af7c8074a6fb9de14102e84f0b9` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:34.216961 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=34658af7c8074a6fb9de14102e84f0b9)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:34.219139 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:34.268450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '34658af7c8074a6fb9de14102e84f0b9'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.282643 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '34658af7c8074a6fb9de14102e84f0b9'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.283372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '34658af7c8074a6fb9de14102e84f0b9'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.284851 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '34658af7c8074a6fb9de14102e84f0b9'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.294647 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '34658af7c8074a6fb9de14102e84f0b9'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.316169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1330/2664] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:34 2022] DELETE /identity/v3/users/34658af7c8074a6fb9de14102e84f0b9 => generated 0 bytes in 306 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:34.320757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:34.321039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:34.321354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:34.343331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:34.344608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:34.407024 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1335/2665] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:34.417192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:34.418986 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:34.491535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00661a3c-0061-41c3-b461-b4420bc8ac6e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:34.495204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:34.495806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:34.496078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:34.496358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:34.506157 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:34.507616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:34.518180 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1331/2666] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:34.792572 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:34.794187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:34.807858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7f4dd1-09b9-427a-9b15-475e6f63b447 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:34.814605 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:34.815316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:34.815562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:34.815830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] PATH_INFO: `/v3/projects/275ae501c31a43d6bb7795288d4df76d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:34.817167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] RBAC: Authorizing `identity:delete_project(project_id=275ae501c31a43d6bb7795288d4df76d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:34.819325 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:34.873615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '275ae501c31a43d6bb7795288d4df76d', 'request_id': 'req-f4a9e92b-3e05-4ab8-a60d-698f4448578f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.882354 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '275ae501c31a43d6bb7795288d4df76d', 'request_id': 'req-f4a9e92b-3e05-4ab8-a60d-698f4448578f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.889621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '275ae501c31a43d6bb7795288d4df76d', 'request_id': 'req-f4a9e92b-3e05-4ab8-a60d-698f4448578f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.900695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] The token cache is being invalidate because project 275ae501c31a43d6bb7795288d4df76d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:34.900971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.901617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:34.903153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1336/2667] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:34 2022] DELETE /identity/v3/projects/275ae501c31a43d6bb7795288d4df76d => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:34.907283 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:34.917455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eda2c776-9b3d-46eb-999d-71b60d197fcd service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:34.965595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:34.976134 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:34.976768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:34.977057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:34.977302 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] PATH_INFO: `/v3/users/1407e2ea43fc494896e317c5e8f64797` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:34.978638 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1407e2ea43fc494896e317c5e8f64797)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:34.980819 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:35.034448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1407e2ea43fc494896e317c5e8f64797'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.041560 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1407e2ea43fc494896e317c5e8f64797'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.042271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1407e2ea43fc494896e317c5e8f64797'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.042866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1407e2ea43fc494896e317c5e8f64797'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.051608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6c36ca14-ccf5-4e4e-bbe7-88bf16431591 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1407e2ea43fc494896e317c5e8f64797'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.072358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1332/2668] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:34 2022] DELETE /identity/v3/users/1407e2ea43fc494896e317c5e8f64797 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:35.393772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:35.395278 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:35.410651 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4a9e92b-3e05-4ab8-a60d-698f4448578f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:35.419768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:35.481397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:35.482502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.483156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.483397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] PATH_INFO: `/v3/projects/6ff5c212574e450087cd2cb774d07d61` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.484659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ff5c212574e450087cd2cb774d07d61)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:35.486850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:35.544977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ff5c212574e450087cd2cb774d07d61', 'request_id': 'req-583a9f24-8a7f-4437-801f-873d192315d3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.555372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ff5c212574e450087cd2cb774d07d61', 'request_id': 'req-583a9f24-8a7f-4437-801f-873d192315d3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.562397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ff5c212574e450087cd2cb774d07d61', 'request_id': 'req-583a9f24-8a7f-4437-801f-873d192315d3'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.573714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] The token cache is being invalidate because project 6ff5c212574e450087cd2cb774d07d61 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:35.574029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.574743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-583a9f24-8a7f-4437-801f-873d192315d3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:35.576474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1337/2669] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:35 2022] DELETE /identity/v3/projects/6ff5c212574e450087cd2cb774d07d61 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:35.581729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12b7d8d-1d9d-4456-80c9-41e683396ef0 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.582025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12b7d8d-1d9d-4456-80c9-41e683396ef0 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.582325 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12b7d8d-1d9d-4456-80c9-41e683396ef0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.603091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a12b7d8d-1d9d-4456-80c9-41e683396ef0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:35.604280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a12b7d8d-1d9d-4456-80c9-41e683396ef0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:35.665395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1333/2670] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:35.674707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7040c019-bc57-446a-bdc7-1e87bf7da483 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.675120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7040c019-bc57-446a-bdc7-1e87bf7da483 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.675555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7040c019-bc57-446a-bdc7-1e87bf7da483 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.699635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7040c019-bc57-446a-bdc7-1e87bf7da483 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:35.700857 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7040c019-bc57-446a-bdc7-1e87bf7da483 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:35.709439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1338/2671] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:35.713938 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12b7d8d-1d9d-4456-80c9-41e683396ef0 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:35.724080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12b7d8d-1d9d-4456-80c9-41e683396ef0 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:35.729192 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:35.729784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.730055 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.730348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.730995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:35.731329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:35.733179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:35.744788 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1334/2672] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:35.749248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7040c019-bc57-446a-bdc7-1e87bf7da483 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:35.758704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7040c019-bc57-446a-bdc7-1e87bf7da483 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:35.764229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:35.764819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.765092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.765341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.765964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:35.767898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:35.802687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1339/2673] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:35 2022] POST /identity/v3/projects => generated 387 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:35.807393 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:35.818218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-076a02f8-5419-4af1-9e3b-b2099afa94be admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:35.879017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:35.879626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.879900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.880163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.880784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:35.882898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:35.891572 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:35.908430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1335/2674] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:35 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:35.912876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:35.924010 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1abdc84d-a0e5-403d-8069-5045b7ab0834 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:35.929317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:35.929866 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.930133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.930404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.931051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:35.932652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:35.943224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1340/2675] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:35 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:35.947504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:35.957564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1d28a4-9df3-4192-a2a0-f419c2ff7c1b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:35.962076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ad29adb-ef0d-48f2-a8f9-db3776c237b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:35.962651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad29adb-ef0d-48f2-a8f9-db3776c237b4 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.962924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad29adb-ef0d-48f2-a8f9-db3776c237b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.963191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ad29adb-ef0d-48f2-a8f9-db3776c237b4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:35.963833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad29adb-ef0d-48f2-a8f9-db3776c237b4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:35.965426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ad29adb-ef0d-48f2-a8f9-db3776c237b4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:35.975942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1336/2676] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:35.980546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:35.991067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53098556-eafb-4832-935e-eebcf76f2792 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:35.997248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:35.998055 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:35.998407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:35.998679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] PATH_INFO: `/v3/projects/b9013253783e4df7a49ba88c6e8d738d/users/0faf1d6adec14fb4a2239940d90aabc1/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.014255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b9013253783e4df7a49ba88c6e8d738d, user_id=0faf1d6adec14fb4a2239940d90aabc1, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.016919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.032501 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1341/2677] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:35 2022] PUT /identity/v3/projects/b9013253783e4df7a49ba88c6e8d738d/users/0faf1d6adec14fb4a2239940d90aabc1/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:36.044178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3d7187-0b1c-4160-a881-1baebdb2b9f4 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.044995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3d7187-0b1c-4160-a881-1baebdb2b9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.044995 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa3d7187-0b1c-4160-a881-1baebdb2b9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.074090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-aa3d7187-0b1c-4160-a881-1baebdb2b9f4 None None] MFA Rules not processed for user `0faf1d6adec14fb4a2239940d90aabc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:36.075453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-aa3d7187-0b1c-4160-a881-1baebdb2b9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:36.152760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1337/2678] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:36 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:36.154057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.196909 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:36.223839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca24c6a1-e078-41d8-b837-1ac8a75264c2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.228922 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.229534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.229811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.230115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.230770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.233170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.265750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1342/2679] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:36 2022] POST /identity/v3/projects => generated 383 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:36.269836 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3d7187-0b1c-4160-a881-1baebdb2b9f4 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.280888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa3d7187-0b1c-4160-a881-1baebdb2b9f4 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.343971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.344581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.344850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.345111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.345762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.347768 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.374497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1338/2680] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:36 2022] POST /identity/v3/users => generated 292 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:36.378791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.390105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa7edad-a8a9-4cdc-bcf3-f4a529bbd0ea admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.395177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.395848 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.396188 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.396526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.397229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.398920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.413080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1343/2681] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:36.415466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.426105 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5043d837-1f17-4b50-9e15-006290de0cd4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.430954 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e923509-ef1f-4d6e-a506-974e44a5f159 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.432420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e923509-ef1f-4d6e-a506-974e44a5f159 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.432703 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e923509-ef1f-4d6e-a506-974e44a5f159 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.432974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e923509-ef1f-4d6e-a506-974e44a5f159 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.433620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e923509-ef1f-4d6e-a506-974e44a5f159 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.435256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e923509-ef1f-4d6e-a506-974e44a5f159 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.445825 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1339/2682] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:36.450309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.460624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d43f12-7409-4a1f-b931-0724f9c5c950 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.465214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.465881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.466186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.466482 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] PATH_INFO: `/v3/projects/35ae4c7dfe914e1fb6d415fc432098d7/users/eab1534939d84a9a909cc4005d2f8c03/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.481664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=35ae4c7dfe914e1fb6d415fc432098d7, user_id=eab1534939d84a9a909cc4005d2f8c03, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.484562 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.503549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1344/2683] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:36 2022] PUT /identity/v3/projects/35ae4c7dfe914e1fb6d415fc432098d7/users/eab1534939d84a9a909cc4005d2f8c03/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:36.515035 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb167f09-889c-4793-b982-d893389aec35 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.515304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb167f09-889c-4793-b982-d893389aec35 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.515749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb167f09-889c-4793-b982-d893389aec35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.554317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-cb167f09-889c-4793-b982-d893389aec35 None None] MFA Rules not processed for user `eab1534939d84a9a909cc4005d2f8c03`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:36.555549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-cb167f09-889c-4793-b982-d893389aec35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:36.595232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:36.627812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1340/2684] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:36.633229 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.706487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c03a55-a3a8-45c5-9c2b-b167b498a2b1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.711751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.712364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.712637 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.712906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.713633 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.715742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.744469 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1345/2685] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:36 2022] POST /identity/v3/projects => generated 385 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:36.748619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb167f09-889c-4793-b982-d893389aec35 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.759543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb167f09-889c-4793-b982-d893389aec35 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.819272 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.819889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.820163 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.820456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.821181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.823207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.852011 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1341/2686] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:36 2022] POST /identity/v3/users => generated 299 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:24:36.856291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.867753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f24e6bf3-121c-4163-9667-3b876718abb9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.873580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.874238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.874511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.874765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.875409 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.877052 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.887879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1346/2687] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:36.892323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.902332 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fae29e-7739-4194-80b1-8aece7963245 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:36.907522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:36.908155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:36.908425 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:36.908692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] PATH_INFO: `/v3/projects/93120d76ec83492e80d5e2dfb49762f8/users/16caee60f1f44e6da06ba49ebdc077ae/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:36.923431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93120d76ec83492e80d5e2dfb49762f8, user_id=16caee60f1f44e6da06ba49ebdc077ae, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:36.926095 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:36.941322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1342/2688] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:36 2022] PUT /identity/v3/projects/93120d76ec83492e80d5e2dfb49762f8/users/16caee60f1f44e6da06ba49ebdc077ae/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:36.945521 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:36.956458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b944049b-3f0d-4b30-8e05-4b18522e622b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:37.016794 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e47c08d7-f8c8-4b3a-9b72-c1d34f8d3c80 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:37.017406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47c08d7-f8c8-4b3a-9b72-c1d34f8d3c80 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:37.017680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47c08d7-f8c8-4b3a-9b72-c1d34f8d3c80 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:37.017948 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e47c08d7-f8c8-4b3a-9b72-c1d34f8d3c80 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:37.018619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47c08d7-f8c8-4b3a-9b72-c1d34f8d3c80 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:37.020190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e47c08d7-f8c8-4b3a-9b72-c1d34f8d3c80 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:37.022333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:37.039849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1347/2689] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:36 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:37.039992 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:37.046596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece06ec-b913-4dff-beab-4cc4b3acfaa2 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:37.052705 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:37.053509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:37.054087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:37.054247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] PATH_INFO: `/v3/projects/93120d76ec83492e80d5e2dfb49762f8/users/16caee60f1f44e6da06ba49ebdc077ae/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:37.056616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] RBAC: Authorizing `identity:create_grant(project_id=93120d76ec83492e80d5e2dfb49762f8, user_id=16caee60f1f44e6da06ba49ebdc077ae, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:37.059586 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:37.071660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1343/2690] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:37 2022] PUT /identity/v3/projects/93120d76ec83492e80d5e2dfb49762f8/users/16caee60f1f44e6da06ba49ebdc077ae/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:37.082153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:37.082438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:37.082710 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:37.112769 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] MFA Rules not processed for user `16caee60f1f44e6da06ba49ebdc077ae`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:37.113994 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:37.193310 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1348/2691] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:37 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:37.210596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:37.212672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:37.273437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73464652-0c15-46d3-8415-e467ea264720 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:37.276945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:37.277545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:37.277813 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:37.278107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:37.288469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:37.289927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:37.300682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1344/2692] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:37 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:37.399695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:37.401053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:37.459320 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3261be0-e38d-4386-b80a-eec08538e3d5 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:37.463473 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:37.464119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:37.464391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:37.464690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:37.466879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:37.539667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:37.541098 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:37.552696 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1349/2693] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:37.866372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:37.867875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:37.880160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-765db2f3-9abe-4019-ae81-85366534aec9 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:37.884226 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a87feac-269c-4731-8909-f49a154e9e7e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:37.884822 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a87feac-269c-4731-8909-f49a154e9e7e service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:37.885096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a87feac-269c-4731-8909-f49a154e9e7e service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:37.885362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a87feac-269c-4731-8909-f49a154e9e7e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:37.916307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:37.948946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a87feac-269c-4731-8909-f49a154e9e7e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:37.950450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a87feac-269c-4731-8909-f49a154e9e7e service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:37.961173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1345/2694] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:37 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:40.956870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:40.958548 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:40.971905 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2907a3c-3a66-4e0e-9e0e-674501cb6026 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:40.977105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:40.977641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:40.978214 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:40.978523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] PATH_INFO: `/v3/users/d0ed4854e0d449e58d09487fe8eedc2c` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:40.979626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] RBAC: Authorizing `identity:delete_user(user_id=d0ed4854e0d449e58d09487fe8eedc2c)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:40.981545 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.031105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:41.051387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd0ed4854e0d449e58d09487fe8eedc2c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.058942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd0ed4854e0d449e58d09487fe8eedc2c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.059611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd0ed4854e0d449e58d09487fe8eedc2c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.060219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd0ed4854e0d449e58d09487fe8eedc2c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.069219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd0ed4854e0d449e58d09487fe8eedc2c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.088728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1350/2695] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:40 2022] DELETE /identity/v3/users/d0ed4854e0d449e58d09487fe8eedc2c => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:41.115802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b13b0a-5078-4300-a066-f900b82c2f43 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.116448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b13b0a-5078-4300-a066-f900b82c2f43 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.116448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b13b0a-5078-4300-a066-f900b82c2f43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.136294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-03b13b0a-5078-4300-a066-f900b82c2f43 None None] MFA Rules not processed for user `d83a16ba92834ab6a23a4b4634711178`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:41.137826 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-03b13b0a-5078-4300-a066-f900b82c2f43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:41.191596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1346/2696] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:41.279257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:41.280916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:41.363916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7e0077-e742-41f0-b9dd-6d378adce070 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:41.369106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:41.369739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.370023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.370339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] PATH_INFO: `/v3/projects/ad99e285720d41988cf2cf8f8978c313` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.371457 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad99e285720d41988cf2cf8f8978c313)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:41.373474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.426309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad99e285720d41988cf2cf8f8978c313', 'request_id': 'req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.438380 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad99e285720d41988cf2cf8f8978c313', 'request_id': 'req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.445850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad99e285720d41988cf2cf8f8978c313', 'request_id': 'req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.456996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] The token cache is being invalidate because project ad99e285720d41988cf2cf8f8978c313 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:41.457279 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.457936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-32bdd4d9-89c9-4fd2-baed-9b27c6f65beb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:41.459474 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1351/2697] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:41 2022] DELETE /identity/v3/projects/ad99e285720d41988cf2cf8f8978c313 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:41.464423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d9153-d275-4b33-adf0-eaefe266ce34 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.464717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d9153-d275-4b33-adf0-eaefe266ce34 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.464978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8d9153-d275-4b33-adf0-eaefe266ce34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.487862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9b8d9153-d275-4b33-adf0-eaefe266ce34 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:41.489335 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9b8d9153-d275-4b33-adf0-eaefe266ce34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:41.562569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1347/2698] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:41.570679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893dddc-2d15-4dd7-b5a8-ffeeb84f0f2c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.570962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893dddc-2d15-4dd7-b5a8-ffeeb84f0f2c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.571237 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893dddc-2d15-4dd7-b5a8-ffeeb84f0f2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.591053 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-5893dddc-2d15-4dd7-b5a8-ffeeb84f0f2c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:41.592317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5893dddc-2d15-4dd7-b5a8-ffeeb84f0f2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:41.600613 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1352/2699] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:41.605148 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d9153-d275-4b33-adf0-eaefe266ce34 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:41.615387 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b8d9153-d275-4b33-adf0-eaefe266ce34 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:41.620365 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:41.621044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.621404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.622027 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.622510 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:41.622861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:41.624911 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.638021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1348/2700] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:24:41.642535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5893dddc-2d15-4dd7-b5a8-ffeeb84f0f2c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:41.651917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5893dddc-2d15-4dd7-b5a8-ffeeb84f0f2c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:41.657599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:41.658233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.658507 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.658805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.659437 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:41.661377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.691566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1353/2701] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:41 2022] POST /identity/v3/projects => generated 373 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:41.695502 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:41.707666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96515a-1cf6-4a7b-8682-7efe19f2cd70 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:41.769472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:41.770144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.770413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.770680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.771306 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:41.773211 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.782702 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:41.798071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1349/2702] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:41 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:41.804274 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:41.816227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753471d0-72f7-4904-9a09-40213acd6c38 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:41.824526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:41.825415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.825979 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.826351 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.828134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:41.830102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.832714 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:41.846288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1354/2703] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:41 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:41.850334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:41.862471 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ec16216-86cd-4927-b075-8521c201c003 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:41.868694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23be1199-7d1f-4905-a389-b6f379f1d98e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:41.869301 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23be1199-7d1f-4905-a389-b6f379f1d98e admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.870141 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23be1199-7d1f-4905-a389-b6f379f1d98e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.870441 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23be1199-7d1f-4905-a389-b6f379f1d98e admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.871125 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23be1199-7d1f-4905-a389-b6f379f1d98e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:41.872791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23be1199-7d1f-4905-a389-b6f379f1d98e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.884753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1350/2704] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:41 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:41.889566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:41.900688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fde2da8c-6047-4723-bbaa-49d602527bdf admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:41.905739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:41.906516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.906795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.907075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] PATH_INFO: `/v3/projects/4aff00d4480c4797a540c1814459219e/users/35e2db687fa1452289605831576cfb99/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.922765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] RBAC: Authorizing `identity:create_grant(project_id=4aff00d4480c4797a540c1814459219e, user_id=35e2db687fa1452289605831576cfb99, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:41.925872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:41.940441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1355/2705] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:41 2022] PUT /identity/v3/projects/4aff00d4480c4797a540c1814459219e/users/35e2db687fa1452289605831576cfb99/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:41.950528 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:41.950800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:41.951062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:41.979688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] MFA Rules not processed for user `35e2db687fa1452289605831576cfb99`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:41.980887 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:42.064977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1351/2706] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:41 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:42.077491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:42.079632 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:42.160618 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f4d6b4a-e323-467a-8baa-d9284f4e975c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:42.166712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:42.167341 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:42.167616 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:42.167893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:42.176846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:42.178644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:42.189594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1356/2707] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:42 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:24:42.269268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:42.271315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:42.340299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92079566-9b8c-4038-8f4f-ddc66e90469b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:42.346357 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed25feb0-ca90-4a67-8195-73909953be29 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:42.347545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25feb0-ca90-4a67-8195-73909953be29 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:42.347840 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25feb0-ca90-4a67-8195-73909953be29 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:42.348104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed25feb0-ca90-4a67-8195-73909953be29 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:42.350464 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ed25feb0-ca90-4a67-8195-73909953be29 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:42.422385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed25feb0-ca90-4a67-8195-73909953be29 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:42.424005 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed25feb0-ca90-4a67-8195-73909953be29 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:42.433778 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1352/2708] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 165 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:44.768932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:44.770382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:44.836030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef69debe-30c4-4761-9ae7-dd324c5adf87 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:44.841000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:44.841396 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:44.841672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:44.841947 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] PATH_INFO: `/v3/users/0faf1d6adec14fb4a2239940d90aabc1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:44.843190 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0faf1d6adec14fb4a2239940d90aabc1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:44.845143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:44.895762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0faf1d6adec14fb4a2239940d90aabc1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:44.903167 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0faf1d6adec14fb4a2239940d90aabc1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:44.903851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0faf1d6adec14fb4a2239940d90aabc1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:44.904465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0faf1d6adec14fb4a2239940d90aabc1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:44.908112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:44.917870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0faf1d6adec14fb4a2239940d90aabc1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:44.938586 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1357/2709] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:44 2022] DELETE /identity/v3/users/0faf1d6adec14fb4a2239940d90aabc1 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:44.948842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:44.948842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:44.948842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:44.950992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:44.950992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:44.965228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:44.966522 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:44.985082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:45.017153 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e661b7-e44f-46ef-8d54-06b39a19a647 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:45.021747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:45.022408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.022680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.022946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] PATH_INFO: `/v3/users/35e2db687fa1452289605831576cfb99` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.024054 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] RBAC: Authorizing `identity:delete_user(user_id=35e2db687fa1452289605831576cfb99)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:45.026048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:45.028280 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1353/2710] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:45.038570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:45.040229 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.074316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '35e2db687fa1452289605831576cfb99'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.081258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '35e2db687fa1452289605831576cfb99'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.082025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '35e2db687fa1452289605831576cfb99'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.082695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '35e2db687fa1452289605831576cfb99'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.091208 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-d67a6fa5-3183-4f5e-b919-1ec947db3800 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '35e2db687fa1452289605831576cfb99'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.098478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee5af1f-e4ca-4755-9128-124af235b1ff None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:45.102077 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:45.102848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.103121 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.103395 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.105130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.108371 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1358/2711] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:44 2022] DELETE /identity/v3/users/35e2db687fa1452289605831576cfb99 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:45.112996 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.113384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.113910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.117361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:45.118860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:45.128411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1354/2712] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:45.133877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:45.135122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.184728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:45.199109 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1359/2713] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:24:45.209331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:45.210898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.224224 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e93ce8-666e-460f-958a-c229b93bc900 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:45.281466 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:45.282317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.282652 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.282839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.292547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:45.293982 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:45.304261 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1355/2714] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:45.505122 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:45.506624 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.518871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a17bba8a-3aa6-40ae-8c57-ee56543b8613 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:45.527778 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:45.528563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.528956 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.529353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] PATH_INFO: `/v3/projects/b9013253783e4df7a49ba88c6e8d738d` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.536041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b9013253783e4df7a49ba88c6e8d738d)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:45.538045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:45.595370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b9013253783e4df7a49ba88c6e8d738d', 'request_id': 'req-ff7505d9-6b61-4705-9711-43c6dcdc9283'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.604264 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b9013253783e4df7a49ba88c6e8d738d', 'request_id': 'req-ff7505d9-6b61-4705-9711-43c6dcdc9283'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.611497 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b9013253783e4df7a49ba88c6e8d738d', 'request_id': 'req-ff7505d9-6b61-4705-9711-43c6dcdc9283'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.622963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] The token cache is being invalidate because project b9013253783e4df7a49ba88c6e8d738d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:45.623333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.624045 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.625571 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1360/2715] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:45 2022] DELETE /identity/v3/projects/b9013253783e4df7a49ba88c6e8d738d => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:45.629698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:45.640615 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96df792e-4750-4250-ba1d-cdc22c1d2cde service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:45.662591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:45.663993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.712597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:45.713264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.713537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.713807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] PATH_INFO: `/v3/users/eab1534939d84a9a909cc4005d2f8c03` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.714987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] RBAC: Authorizing `identity:delete_user(user_id=eab1534939d84a9a909cc4005d2f8c03)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:45.717062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:45.731717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:45.732058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff7505d9-6b61-4705-9711-43c6dcdc9283 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:45.737327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:45.737964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.738272 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.738540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] PATH_INFO: `/v3/projects/4aff00d4480c4797a540c1814459219e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.739751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] RBAC: Authorizing `identity:delete_project(project_id=4aff00d4480c4797a540c1814459219e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:45.741648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:45.771253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eab1534939d84a9a909cc4005d2f8c03'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.778847 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eab1534939d84a9a909cc4005d2f8c03'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.779489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eab1534939d84a9a909cc4005d2f8c03'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.780118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eab1534939d84a9a909cc4005d2f8c03'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.788433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-989e5cf2-1cee-475c-9830-dba5e5f9ba8a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eab1534939d84a9a909cc4005d2f8c03'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.793120 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4aff00d4480c4797a540c1814459219e', 'request_id': 'req-083deb28-8868-4e77-9e1a-0bb0319d998c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.803128 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4aff00d4480c4797a540c1814459219e', 'request_id': 'req-083deb28-8868-4e77-9e1a-0bb0319d998c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.807007 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1356/2716] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:45 2022] DELETE /identity/v3/users/eab1534939d84a9a909cc4005d2f8c03 => generated 0 bytes in 178 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:45.809962 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4aff00d4480c4797a540c1814459219e', 'request_id': 'req-083deb28-8868-4e77-9e1a-0bb0319d998c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.822198 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] The token cache is being invalidate because project 4aff00d4480c4797a540c1814459219e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:45.822485 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.823168 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-083deb28-8868-4e77-9e1a-0bb0319d998c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:45.824811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1361/2717] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:45 2022] DELETE /identity/v3/projects/4aff00d4480c4797a540c1814459219e => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:45.829984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553d77ef-9d65-4170-842c-9cadc7111524 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.830329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553d77ef-9d65-4170-842c-9cadc7111524 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.830717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553d77ef-9d65-4170-842c-9cadc7111524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.850248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-553d77ef-9d65-4170-842c-9cadc7111524 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:45.851447 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-553d77ef-9d65-4170-842c-9cadc7111524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.914053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1357/2718] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:45.921887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a0fcb6-1bb1-4818-8531-4cc0339a6563 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.922200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a0fcb6-1bb1-4818-8531-4cc0339a6563 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.922478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a0fcb6-1bb1-4818-8531-4cc0339a6563 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.942995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-c1a0fcb6-1bb1-4818-8531-4cc0339a6563 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:45.944225 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c1a0fcb6-1bb1-4818-8531-4cc0339a6563 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:45.952515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1362/2719] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:45.957010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553d77ef-9d65-4170-842c-9cadc7111524 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:45.969571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553d77ef-9d65-4170-842c-9cadc7111524 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:45.975299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:45.976394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:45.976394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:45.976394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:45.977044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:45.977339 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:45.979255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:45.992204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1358/2720] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:45.996636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a0fcb6-1bb1-4818-8531-4cc0339a6563 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.002081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:46.007233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a0fcb6-1bb1-4818-8531-4cc0339a6563 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.013125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.013985 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.014998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.015277 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.015972 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.018178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.048178 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1363/2721] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:45 2022] POST /identity/v3/projects => generated 367 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:46.052503 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.062721 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d16a20-e975-4553-9fb1-e3bdfdd28f53 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.089916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.091774 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:46.130899 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.131542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.131812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.132076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.132712 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.134688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.158080 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1359/2722] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:46 2022] POST /identity/v3/users => generated 284 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:46.161953 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.166227 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85786416-5154-4ffa-8558-2d412b0c38c9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.171254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.171892 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.172172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.172445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] PATH_INFO: `/v3/projects/35ae4c7dfe914e1fb6d415fc432098d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.173579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] RBAC: Authorizing `identity:delete_project(project_id=35ae4c7dfe914e1fb6d415fc432098d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.174498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d18c8ae5-2b5d-461a-a7bd-c9209535d2c2 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.175532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.179188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.179791 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.180046 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.180309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.180943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.182639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.183947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:46.194671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1360/2723] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:46 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:24:46.198671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.208644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95029a5-314f-4a39-a7e0-a500d37af71d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.213411 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.214241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.214512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.214777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.215407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.217092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.226305 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '35ae4c7dfe914e1fb6d415fc432098d7', 'request_id': 'req-be4fb41c-373f-4be3-a532-169900762c05'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.227324 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1361/2724] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:46 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:24:46.231247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.237425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '35ae4c7dfe914e1fb6d415fc432098d7', 'request_id': 'req-be4fb41c-373f-4be3-a532-169900762c05'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.241193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f59994c-1f29-474d-b50e-d2ed0ac21221 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.244525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '35ae4c7dfe914e1fb6d415fc432098d7', 'request_id': 'req-be4fb41c-373f-4be3-a532-169900762c05'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.255173 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] The token cache is being invalidate because project 35ae4c7dfe914e1fb6d415fc432098d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:46.255519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.256197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.257731 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1364/2725] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:46 2022] DELETE /identity/v3/projects/35ae4c7dfe914e1fb6d415fc432098d7 => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:46.261673 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.263453 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:46.297220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c5aeb8f-d1b6-49ea-bc38-50c3ef52798b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.297879 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5aeb8f-d1b6-49ea-bc38-50c3ef52798b admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.298178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5aeb8f-d1b6-49ea-bc38-50c3ef52798b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.298497 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5aeb8f-d1b6-49ea-bc38-50c3ef52798b admin admin] PATH_INFO: `/v3/projects/bdb1223eca3f4a0ea059a2cb613d4068/users/ff60d527fc6f47e59747d7776038d2f6/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.313337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5aeb8f-d1b6-49ea-bc38-50c3ef52798b admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdb1223eca3f4a0ea059a2cb613d4068, user_id=ff60d527fc6f47e59747d7776038d2f6, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.316088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c5aeb8f-d1b6-49ea-bc38-50c3ef52798b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.329850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1362/2726] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:46 2022] PUT /identity/v3/projects/bdb1223eca3f4a0ea059a2cb613d4068/users/ff60d527fc6f47e59747d7776038d2f6/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:46.330963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be4fb41c-373f-4be3-a532-169900762c05 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.341299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.341578 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.341862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.346436 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:46.368829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] MFA Rules not processed for user `ff60d527fc6f47e59747d7776038d2f6`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:46.370156 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:46.394789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.395431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.395707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.395980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] PATH_INFO: `/v3/users/16caee60f1f44e6da06ba49ebdc077ae` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.397134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] RBAC: Authorizing `identity:delete_user(user_id=16caee60f1f44e6da06ba49ebdc077ae)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.399114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.438786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1363/2727] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:46 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:24:46.450460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:46.452512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:46.519099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5d2d9f-724b-4ba1-b70c-c96575bfd2f5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:46.522804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:46.523430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:46.523701 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:46.523969 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:46.533956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:46.535456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:46.542103 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:46.546977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1364/2728] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:46 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:46.580205 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '16caee60f1f44e6da06ba49ebdc077ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.587566 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '16caee60f1f44e6da06ba49ebdc077ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.588377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '16caee60f1f44e6da06ba49ebdc077ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.589626 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '16caee60f1f44e6da06ba49ebdc077ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.600516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-de5355c3-8304-4d10-a201-5ee4799d386b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '16caee60f1f44e6da06ba49ebdc077ae'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:46.626992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1365/2729] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:46 2022] DELETE /identity/v3/users/16caee60f1f44e6da06ba49ebdc077ae => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:47.006797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.019000 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-103ae52b-c02a-4a01-9852-0c2773800403 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.082898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.083439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.083710 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.083976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] PATH_INFO: `/v3/projects/93120d76ec83492e80d5e2dfb49762f8` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.085144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=93120d76ec83492e80d5e2dfb49762f8)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.087124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.138320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '93120d76ec83492e80d5e2dfb49762f8', 'request_id': 'req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:47.148963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '93120d76ec83492e80d5e2dfb49762f8', 'request_id': 'req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:47.156310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '93120d76ec83492e80d5e2dfb49762f8', 'request_id': 'req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:47.166965 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] The token cache is being invalidate because project 93120d76ec83492e80d5e2dfb49762f8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:47.167254 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:47.167902 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1bbcc1ad-c9d8-4ebc-b16c-c975bb4c8c58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:47.169452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1365/2730] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:47 2022] DELETE /identity/v3/projects/93120d76ec83492e80d5e2dfb49762f8 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:47.176142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e32d631-0ee8-4efc-bdad-7a212ef3ec4b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.176664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e32d631-0ee8-4efc-bdad-7a212ef3ec4b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.176664 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e32d631-0ee8-4efc-bdad-7a212ef3ec4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.197753 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7e32d631-0ee8-4efc-bdad-7a212ef3ec4b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:47.199362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7e32d631-0ee8-4efc-bdad-7a212ef3ec4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:47.260389 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1366/2731] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:47.270189 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8036d3-8d67-4626-a6b7-324be54e3823 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.270475 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8036d3-8d67-4626-a6b7-324be54e3823 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.270809 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a8036d3-8d67-4626-a6b7-324be54e3823 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.292347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7a8036d3-8d67-4626-a6b7-324be54e3823 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:47.293588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7a8036d3-8d67-4626-a6b7-324be54e3823 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:47.302734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1366/2732] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:47.307500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e32d631-0ee8-4efc-bdad-7a212ef3ec4b None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.318709 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e32d631-0ee8-4efc-bdad-7a212ef3ec4b None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.323472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.324074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.324346 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.324611 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.325251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:47.325532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.327856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.344772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1367/2733] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:47.349303 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8036d3-8d67-4626-a6b7-324be54e3823 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.359636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a8036d3-8d67-4626-a6b7-324be54e3823 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.364792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.365448 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.365717 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.365988 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.366728 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.368628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.397388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1367/2734] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:47 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:47.401625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.412152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b59237-61c3-4738-a4aa-9df3b2c32c7e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.432817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:47.471260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.471915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.472204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.472475 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.473140 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.475112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.502037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1368/2735] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:47 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:47.506553 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.517811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bad03f1-d24a-4da8-91df-9eba85a00365 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.523668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.524336 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.524608 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.524894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.525634 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.527804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.538888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1368/2736] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:47 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:47.543044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.553446 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33b30f83-099f-469a-8fb6-f83715fd7960 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.559065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e875c860-4f26-4dea-b2ca-d6d0041b7ed2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.559730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875c860-4f26-4dea-b2ca-d6d0041b7ed2 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.560040 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875c860-4f26-4dea-b2ca-d6d0041b7ed2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.560296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e875c860-4f26-4dea-b2ca-d6d0041b7ed2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.561022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e875c860-4f26-4dea-b2ca-d6d0041b7ed2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.563321 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e875c860-4f26-4dea-b2ca-d6d0041b7ed2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.574532 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1369/2737] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:47 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:47.579122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.589551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa6da0f-fb32-4612-8733-e50072bbe538 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.595178 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.595772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.596051 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.596323 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] PATH_INFO: `/v3/projects/3968c76fd4464c4dab0977f88694b3e1/users/99a362c0101540e997c3d39de297f502/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.611489 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3968c76fd4464c4dab0977f88694b3e1, user_id=99a362c0101540e997c3d39de297f502, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.614185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.631161 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1369/2738] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:47 2022] PUT /identity/v3/projects/3968c76fd4464c4dab0977f88694b3e1/users/99a362c0101540e997c3d39de297f502/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:47.643508 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.643766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.644063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.674275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] MFA Rules not processed for user `99a362c0101540e997c3d39de297f502`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:47.675523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:47.749603 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1370/2739] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:47 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:47.761611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:47.763871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:47.798074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:47.835696 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6b1722c-7f17-47ee-806e-1d04cf09b1b1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:47.840093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2fa089d-c524-49c4-aa5f-e0e0c716b80d service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd6cfd5241a0147d9bfe0b7addb3e1851', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:47.840748 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa089d-c524-49c4-aa5f-e0e0c716b80d service glance] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:47.841366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa089d-c524-49c4-aa5f-e0e0c716b80d service glance] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:47.841619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2fa089d-c524-49c4-aa5f-e0e0c716b80d service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:47.852996 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa089d-c524-49c4-aa5f-e0e0c716b80d service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:47.854527 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2fa089d-c524-49c4-aa5f-e0e0c716b80d service glance] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:47.876536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1370/2740] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:47 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:55.226739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:55.228387 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:55.299067 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46b2d064-f63b-45b3-a92f-4b268f7780d0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:55.304142 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:55.304766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:55.305033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:55.305299 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] PATH_INFO: `/v3/users/ff60d527fc6f47e59747d7776038d2f6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:55.306523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff60d527fc6f47e59747d7776038d2f6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:55.308439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:55.354984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:55.371245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff60d527fc6f47e59747d7776038d2f6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:55.380230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff60d527fc6f47e59747d7776038d2f6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:55.380788 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff60d527fc6f47e59747d7776038d2f6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:55.381422 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff60d527fc6f47e59747d7776038d2f6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:55.393301 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff60d527fc6f47e59747d7776038d2f6'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:55.416064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1371/2741] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:55 2022] DELETE /identity/v3/users/ff60d527fc6f47e59747d7776038d2f6 => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:55.420695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:55.421205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:55.421316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:55.441845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:55.443150 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:55.508726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1371/2742] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:55.519114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:55.520766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:55.584840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b72619-ed79-4f27-a595-93793d3aecca admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:55.589160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb123b47-025d-40ab-b8cb-2561310d8978 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:55.589841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb123b47-025d-40ab-b8cb-2561310d8978 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:55.590146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb123b47-025d-40ab-b8cb-2561310d8978 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:55.590483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb123b47-025d-40ab-b8cb-2561310d8978 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:55.600677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb123b47-025d-40ab-b8cb-2561310d8978 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:55.602281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb123b47-025d-40ab-b8cb-2561310d8978 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:55.612993 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1372/2743] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:55.925182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:55.926883 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:55.948540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb17d26-2c6a-4eb2-9014-77cf4780c841 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:55.957423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:55.958315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:55.958614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:55.958855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] PATH_INFO: `/v3/projects/bdb1223eca3f4a0ea059a2cb613d4068` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:55.960009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdb1223eca3f4a0ea059a2cb613d4068)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:55.962032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.017661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdb1223eca3f4a0ea059a2cb613d4068', 'request_id': 'req-9cd347fa-32f1-4387-a540-a39a5a19e7ee'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:56.027538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdb1223eca3f4a0ea059a2cb613d4068', 'request_id': 'req-9cd347fa-32f1-4387-a540-a39a5a19e7ee'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:56.035366 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdb1223eca3f4a0ea059a2cb613d4068', 'request_id': 'req-9cd347fa-32f1-4387-a540-a39a5a19e7ee'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:56.046740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] The token cache is being invalidate because project bdb1223eca3f4a0ea059a2cb613d4068 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:56.047049 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:56.047816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9cd347fa-32f1-4387-a540-a39a5a19e7ee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:56.049767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1372/2744] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:55 2022] DELETE /identity/v3/projects/bdb1223eca3f4a0ea059a2cb613d4068 => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:56.055085 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44a552e-9c50-4d07-9453-e29e20b5b7b4 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.055368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44a552e-9c50-4d07-9453-e29e20b5b7b4 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.055641 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a44a552e-9c50-4d07-9453-e29e20b5b7b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.075951 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a44a552e-9c50-4d07-9453-e29e20b5b7b4 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:56.077143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a44a552e-9c50-4d07-9453-e29e20b5b7b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:56.147917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1373/2745] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:56.162386 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44f7fa-932d-4a65-bb73-cb21846a39aa None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.162722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44f7fa-932d-4a65-bb73-cb21846a39aa None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.163110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac44f7fa-932d-4a65-bb73-cb21846a39aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.188429 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-ac44f7fa-932d-4a65-bb73-cb21846a39aa None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:56.189632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ac44f7fa-932d-4a65-bb73-cb21846a39aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:56.199888 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1373/2746] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:56.204334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44a552e-9c50-4d07-9453-e29e20b5b7b4 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.214218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a44a552e-9c50-4d07-9453-e29e20b5b7b4 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.219182 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.219779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.220051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.220319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.220961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:56.221260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.222886 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.236730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1374/2747] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:24:56.241159 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44f7fa-932d-4a65-bb73-cb21846a39aa None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.250667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac44f7fa-932d-4a65-bb73-cb21846a39aa None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.255834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.256418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.256686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.256948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.257600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.259529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.290927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1374/2748] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:56 2022] POST /identity/v3/projects => generated 371 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:56.295034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.305092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-498d9390-cb5c-4eeb-add9-2b7d618532e9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.355317 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:56.377331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.378023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.378313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.378577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.379202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.381236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.405862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1375/2749] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:56 2022] POST /identity/v3/users => generated 286 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:56.410114 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.423970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b5f0f3d-0302-4d56-998b-359363c9f3fd admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.430328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.430932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.431186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.431627 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.432268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.434456 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.445864 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1375/2750] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:56 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:56.450478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.464805 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e3d867-f552-4163-b64a-9566bf1501ab admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.469643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d316686f-e71e-48a4-907b-8389de5f7728 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.470318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316686f-e71e-48a4-907b-8389de5f7728 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.470798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316686f-e71e-48a4-907b-8389de5f7728 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.470798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d316686f-e71e-48a4-907b-8389de5f7728 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.471478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d316686f-e71e-48a4-907b-8389de5f7728 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.473683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d316686f-e71e-48a4-907b-8389de5f7728 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.486898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1376/2751] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:56 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:56.492122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.503852 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94bbaf2-13b7-404d-a4a3-45752724a70b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.509389 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.510081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.510343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.510617 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] PATH_INFO: `/v3/projects/021777a859d5470fa1f192c16c9ac845/users/699fc1f462d342729119656af7db9bba/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.523596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:56.528858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] RBAC: Authorizing `identity:create_grant(project_id=021777a859d5470fa1f192c16c9ac845, user_id=699fc1f462d342729119656af7db9bba, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.532758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.553069 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1376/2752] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:56 2022] PUT /identity/v3/projects/021777a859d5470fa1f192c16c9ac845/users/699fc1f462d342729119656af7db9bba/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:56.563882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9122a31b-fe66-4b2e-8914-6be450cc715c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.564143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9122a31b-fe66-4b2e-8914-6be450cc715c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.564417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9122a31b-fe66-4b2e-8914-6be450cc715c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.594101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-9122a31b-fe66-4b2e-8914-6be450cc715c None None] MFA Rules not processed for user `699fc1f462d342729119656af7db9bba`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:56.595839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9122a31b-fe66-4b2e-8914-6be450cc715c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:56.674720 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1377/2753] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:56 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:56.679774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.744419 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9bb5994-d47b-4d31-9fef-e5a929013e41 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.749439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.750047 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.750348 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.750614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.751235 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.753149 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.785891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1377/2754] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:56 2022] POST /identity/v3/projects => generated 371 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:56.789406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9122a31b-fe66-4b2e-8914-6be450cc715c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.800921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9122a31b-fe66-4b2e-8914-6be450cc715c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.865013 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.865685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.865946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.866253 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.866891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.868793 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.897034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1378/2755] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:56 2022] POST /identity/v3/users => generated 292 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:56.901484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.912430 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-806b1d6c-451b-44ec-8d31-96459ae132b0 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.918096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.918741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.919019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.919285 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.919947 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.921581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.933039 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1378/2756] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:56 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:56.938025 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:56.949434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be7dd9de-3458-4115-8b25-786168ae686d admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:56.954963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:56.955621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:56.955894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:56.956160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] PATH_INFO: `/v3/projects/8fab9d6088124f1bb953c417fc9c4ad6/users/8fc5ae8e341f45c4ab10bf026c56fb65/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:56.972033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fab9d6088124f1bb953c417fc9c4ad6, user_id=8fc5ae8e341f45c4ab10bf026c56fb65, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:56.974836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:56.992506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1379/2757] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:56 2022] PUT /identity/v3/projects/8fab9d6088124f1bb953c417fc9c4ad6/users/8fc5ae8e341f45c4ab10bf026c56fb65/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:24:56.999190 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:57.013871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f82c018-7616-47a1-ba02-e10387d998cd admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:57.078467 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549eeab5-1093-4664-bd73-8e98fd654f32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:57.079082 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549eeab5-1093-4664-bd73-8e98fd654f32 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:57.079353 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549eeab5-1093-4664-bd73-8e98fd654f32 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:57.079618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549eeab5-1093-4664-bd73-8e98fd654f32 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:57.080253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549eeab5-1093-4664-bd73-8e98fd654f32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:57.081834 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549eeab5-1093-4664-bd73-8e98fd654f32 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:57.094028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1379/2758] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:56 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:57.096736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:57.108039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31092e57-5fca-4559-a65a-583d409c8576 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:57.112876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:57.113748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:57.114283 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:57.114627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] PATH_INFO: `/v3/projects/8fab9d6088124f1bb953c417fc9c4ad6/users/8fc5ae8e341f45c4ab10bf026c56fb65/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:57.116662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8fab9d6088124f1bb953c417fc9c4ad6, user_id=8fc5ae8e341f45c4ab10bf026c56fb65, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:57.119418 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:57.131517 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1380/2759] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:57 2022] PUT /identity/v3/projects/8fab9d6088124f1bb953c417fc9c4ad6/users/8fc5ae8e341f45c4ab10bf026c56fb65/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:57.143399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e17c89-e5ad-4aec-8f71-58d80a4c7da1 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:57.143690 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e17c89-e5ad-4aec-8f71-58d80a4c7da1 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:57.143977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e17c89-e5ad-4aec-8f71-58d80a4c7da1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:57.173258 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-90e17c89-e5ad-4aec-8f71-58d80a4c7da1 None None] MFA Rules not processed for user `8fc5ae8e341f45c4ab10bf026c56fb65`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:57.174517 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-90e17c89-e5ad-4aec-8f71-58d80a4c7da1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:57.226209 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:57.248443 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1380/2760] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:57 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:57.260204 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:57.261915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:57.267707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:57.326851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2d15d97-4e35-4ff3-8293-66e1f258d545 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:57.330585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a510de9-d2a7-4b60-9bb2-33d31b5310e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:57.331249 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a510de9-d2a7-4b60-9bb2-33d31b5310e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:57.331588 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a510de9-d2a7-4b60-9bb2-33d31b5310e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:57.331897 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a510de9-d2a7-4b60-9bb2-33d31b5310e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:57.343542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a510de9-d2a7-4b60-9bb2-33d31b5310e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:57.345777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a510de9-d2a7-4b60-9bb2-33d31b5310e5 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:57.356487 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1381/2761] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:57 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:57.783808 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e17c89-e5ad-4aec-8f71-58d80a4c7da1 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:57.850078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e17c89-e5ad-4aec-8f71-58d80a4c7da1 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:57.855628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:57.856240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:57.856511 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:57.856774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] PATH_INFO: `/v3/users/699fc1f462d342729119656af7db9bba` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:57.857950 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=699fc1f462d342729119656af7db9bba)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:57.859914 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:57.912385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '699fc1f462d342729119656af7db9bba'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:57.920036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '699fc1f462d342729119656af7db9bba'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:57.920776 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '699fc1f462d342729119656af7db9bba'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:57.921628 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '699fc1f462d342729119656af7db9bba'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:57.935526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '699fc1f462d342729119656af7db9bba'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:57.953793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1381/2762] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:57 2022] DELETE /identity/v3/users/699fc1f462d342729119656af7db9bba => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:57.958786 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:57.959066 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:57.959412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:57.978630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:57.979835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:58.040768 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1382/2763] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:24:58.051498 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:58.052797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:58.116090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf21018-dd9e-4a3d-9379-ce7cf549c4bf admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:58.121973 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:58.122932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:58.123344 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:58.123775 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:58.135691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:58.137131 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:58.148857 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1382/2764] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:24:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:24:58.439438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:58.442426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:58.455047 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70410ee4-59ea-47b3-8922-5345bf811c0f None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:58.461685 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:58.462450 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:58.462798 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:58.463133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] PATH_INFO: `/v3/projects/021777a859d5470fa1f192c16c9ac845` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:58.464929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] RBAC: Authorizing `identity:delete_project(project_id=021777a859d5470fa1f192c16c9ac845)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:58.467038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:58.518149 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '021777a859d5470fa1f192c16c9ac845', 'request_id': 'req-1e4a9684-6549-4ff2-89e0-60f898bed61f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.528458 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '021777a859d5470fa1f192c16c9ac845', 'request_id': 'req-1e4a9684-6549-4ff2-89e0-60f898bed61f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.536046 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '021777a859d5470fa1f192c16c9ac845', 'request_id': 'req-1e4a9684-6549-4ff2-89e0-60f898bed61f'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.547469 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] The token cache is being invalidate because project 021777a859d5470fa1f192c16c9ac845 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:58.547760 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.548509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.550038 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1383/2765] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:58 2022] DELETE /identity/v3/projects/021777a859d5470fa1f192c16c9ac845 => generated 0 bytes in 111 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:58.554318 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:58.564230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c7bbf3-6f39-4827-bf92-707e78686e20 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:58.569281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:58.625459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:58.625993 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:58.626298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:58.626565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] PATH_INFO: `/v3/users/8fc5ae8e341f45c4ab10bf026c56fb65` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:58.627818 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] RBAC: Authorizing `identity:delete_user(user_id=8fc5ae8e341f45c4ab10bf026c56fb65)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:58.630382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:58.684004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8fc5ae8e341f45c4ab10bf026c56fb65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.691205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8fc5ae8e341f45c4ab10bf026c56fb65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.691904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8fc5ae8e341f45c4ab10bf026c56fb65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.692933 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8fc5ae8e341f45c4ab10bf026c56fb65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.701816 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-b448886d-a3b1-4f8e-9f11-4f38a8c7c39b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8fc5ae8e341f45c4ab10bf026c56fb65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:58.720685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1383/2766] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:24:58 2022] DELETE /identity/v3/users/8fc5ae8e341f45c4ab10bf026c56fb65 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:59.011261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.012706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:59.027090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a9684-6549-4ff2-89e0-60f898bed61f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.056491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:59.090390 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.091034 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.091312 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.091583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] PATH_INFO: `/v3/projects/8fab9d6088124f1bb953c417fc9c4ad6` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.092804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] RBAC: Authorizing `identity:delete_project(project_id=8fab9d6088124f1bb953c417fc9c4ad6)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.094777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.153009 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8fab9d6088124f1bb953c417fc9c4ad6', 'request_id': 'req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:59.162511 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8fab9d6088124f1bb953c417fc9c4ad6', 'request_id': 'req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:59.168736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8fab9d6088124f1bb953c417fc9c4ad6', 'request_id': 'req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:59.178835 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] The token cache is being invalidate because project 8fab9d6088124f1bb953c417fc9c4ad6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:24:59.179117 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:59.179717 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9ed3c756-491d-4b9e-9ea1-f52dd4d0feca admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:24:59.181462 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1384/2767] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:24:59 2022] DELETE /identity/v3/projects/8fab9d6088124f1bb953c417fc9c4ad6 => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:59.188076 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efde1868-ddda-42b0-a458-46fc4bcfb57a None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.188623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efde1868-ddda-42b0-a458-46fc4bcfb57a None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.188623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efde1868-ddda-42b0-a458-46fc4bcfb57a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.208810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-efde1868-ddda-42b0-a458-46fc4bcfb57a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:59.209987 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-efde1868-ddda-42b0-a458-46fc4bcfb57a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:59.267349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1384/2768] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:59.275328 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b8ecd-445d-492d-8324-0a370cecf49a None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.275608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b8ecd-445d-492d-8324-0a370cecf49a None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.275901 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce6b8ecd-445d-492d-8324-0a370cecf49a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.295939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ce6b8ecd-445d-492d-8324-0a370cecf49a None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:59.297163 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ce6b8ecd-445d-492d-8324-0a370cecf49a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:59.305500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1385/2769] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:59.311506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efde1868-ddda-42b0-a458-46fc4bcfb57a None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.321845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efde1868-ddda-42b0-a458-46fc4bcfb57a None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.327432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.328030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.328298 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.328559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.329194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:24:59.329488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.331152 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.343556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1385/2770] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:24:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:24:59.347906 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b8ecd-445d-492d-8324-0a370cecf49a None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.357372 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce6b8ecd-445d-492d-8324-0a370cecf49a None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.362719 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.363307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.363579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.363862 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.364600 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.366622 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.397468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1386/2771] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:59 2022] POST /identity/v3/projects => generated 385 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:59.401610 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.411944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79554f03-dd54-4961-983a-0f39e8f2e5be admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.420592 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:59.475053 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.475651 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.475924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.476193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.476814 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.478780 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.502905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1386/2772] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:59 2022] POST /identity/v3/users => generated 293 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:59.507393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.518362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e8d922-e4ac-4072-b495-68a2d447549e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.523488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.524090 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.524412 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.524687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.525353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.527022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.537472 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1387/2773] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:59.541802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.552282 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9a1a015-f2a6-4846-a1af-1fe1e5b03025 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.557167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae411bc-3e4d-4d7c-b0b6-cc3b8a119600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.557746 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae411bc-3e4d-4d7c-b0b6-cc3b8a119600 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.558030 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae411bc-3e4d-4d7c-b0b6-cc3b8a119600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.558304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae411bc-3e4d-4d7c-b0b6-cc3b8a119600 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.558941 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae411bc-3e4d-4d7c-b0b6-cc3b8a119600 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.560529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae411bc-3e4d-4d7c-b0b6-cc3b8a119600 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.572135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1387/2774] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:59 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:24:59.576564 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.587029 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8362ff4c-cd01-4e6f-9a66-90b12121f34c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.592044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.592694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.592971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.593246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] PATH_INFO: `/v3/projects/abe975f4c06944af8951be322f50858e/users/e73f6c0fa6f24f9a8cf45ea9cb981991/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.608510 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] RBAC: Authorizing `identity:create_grant(project_id=abe975f4c06944af8951be322f50858e, user_id=e73f6c0fa6f24f9a8cf45ea9cb981991, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.611186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.627180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1388/2775] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:59 2022] PUT /identity/v3/projects/abe975f4c06944af8951be322f50858e/users/e73f6c0fa6f24f9a8cf45ea9cb981991/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:24:59.637736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f0d80-0a03-4e15-9a13-0a226baa8317 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.638032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f0d80-0a03-4e15-9a13-0a226baa8317 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.638359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062f0d80-0a03-4e15-9a13-0a226baa8317 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.668488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-062f0d80-0a03-4e15-9a13-0a226baa8317 None None] MFA Rules not processed for user `e73f6c0fa6f24f9a8cf45ea9cb981991`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:24:59.669598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-062f0d80-0a03-4e15-9a13-0a226baa8317 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:24:59.741926 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1388/2776] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:24:59 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:24:59.746666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.807105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:24:59.812830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38071ed5-98da-4717-b1fd-126f8094cc05 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.819627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.820405 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.820912 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.821343 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.822071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.824196 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.853494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1389/2777] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:24:59 2022] POST /identity/v3/projects => generated 385 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:59.857680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062f0d80-0a03-4e15-9a13-0a226baa8317 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.869186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062f0d80-0a03-4e15-9a13-0a226baa8317 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.928233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.928760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.929029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.929293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.929942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.931886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.954905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1389/2778] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:24:59 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:24:59.959296 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:24:59.969958 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f09eeda-ca01-47cc-80a8-579253452e05 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:24:59.975694 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:24:59.976298 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:24:59.976692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:24:59.976965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:24:59.977608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:24:59.979260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:24:59.990842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1390/2779] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:24:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:24:59.995997 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:00.006769 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ac73f70-5683-491f-8542-52247521780e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:00.011866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:00.012523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:00.012796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:00.013067 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] PATH_INFO: `/v3/projects/5f5815fa4b2145f4a77525bdaed8b991/users/e4dac56cadc4463cb2c88db3ece9d2ad/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:00.023088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:00.030928 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f5815fa4b2145f4a77525bdaed8b991, user_id=e4dac56cadc4463cb2c88db3ece9d2ad, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:00.033697 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:00.049170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1390/2780] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:24:59 2022] PUT /identity/v3/projects/5f5815fa4b2145f4a77525bdaed8b991/users/e4dac56cadc4463cb2c88db3ece9d2ad/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:00.053413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:00.063538 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82641fb1-d583-456c-8a07-f2df0dff8fc4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:00.119825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5938d8fd-ad95-4932-98e1-bdc5fec4a8e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:00.120420 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5938d8fd-ad95-4932-98e1-bdc5fec4a8e8 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:00.120691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5938d8fd-ad95-4932-98e1-bdc5fec4a8e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:00.121540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5938d8fd-ad95-4932-98e1-bdc5fec4a8e8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:00.122245 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5938d8fd-ad95-4932-98e1-bdc5fec4a8e8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:00.123850 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5938d8fd-ad95-4932-98e1-bdc5fec4a8e8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:00.134535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1391/2781] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:00 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:00.139295 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:00.150025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-facfdbd3-880a-4f52-9e9a-b0b629298dab admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:00.155185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:00.155823 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:00.156090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:00.156354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] PATH_INFO: `/v3/projects/5f5815fa4b2145f4a77525bdaed8b991/users/e4dac56cadc4463cb2c88db3ece9d2ad/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:00.158312 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5f5815fa4b2145f4a77525bdaed8b991, user_id=e4dac56cadc4463cb2c88db3ece9d2ad, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:00.161045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:00.171770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1391/2782] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:00 2022] PUT /identity/v3/projects/5f5815fa4b2145f4a77525bdaed8b991/users/e4dac56cadc4463cb2c88db3ece9d2ad/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:00.183746 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478c590-b9f3-43be-a7d7-d27835d5ff0c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:00.183914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478c590-b9f3-43be-a7d7-d27835d5ff0c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:00.184330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a478c590-b9f3-43be-a7d7-d27835d5ff0c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:00.221311 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-a478c590-b9f3-43be-a7d7-d27835d5ff0c None None] MFA Rules not processed for user `e4dac56cadc4463cb2c88db3ece9d2ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:00.222591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a478c590-b9f3-43be-a7d7-d27835d5ff0c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:00.293635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1392/2783] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:00 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:00.305088 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:00.307073 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:00.371533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d234aa-c7f0-42dd-9d96-cf0140bb2ee8 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:00.375601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccac34b-e631-4c7a-b881-e1f6f4225d6a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:00.376225 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccac34b-e631-4c7a-b881-e1f6f4225d6a service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:00.376469 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccac34b-e631-4c7a-b881-e1f6f4225d6a service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:00.376719 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccac34b-e631-4c7a-b881-e1f6f4225d6a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:00.385837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccac34b-e631-4c7a-b881-e1f6f4225d6a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:00.387328 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccac34b-e631-4c7a-b881-e1f6f4225d6a service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:00.397355 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1392/2784] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:00 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:00.997103 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478c590-b9f3-43be-a7d7-d27835d5ff0c None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:01.050417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:01.064868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a478c590-b9f3-43be-a7d7-d27835d5ff0c None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:01.070937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:01.071675 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:01.072082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:01.072440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] PATH_INFO: `/v3/users/e73f6c0fa6f24f9a8cf45ea9cb981991` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:01.073659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e73f6c0fa6f24f9a8cf45ea9cb981991)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:01.075609 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:01.132546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e73f6c0fa6f24f9a8cf45ea9cb981991'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:01.140100 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e73f6c0fa6f24f9a8cf45ea9cb981991'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:01.140877 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e73f6c0fa6f24f9a8cf45ea9cb981991'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:01.141516 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e73f6c0fa6f24f9a8cf45ea9cb981991'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:01.150370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e73f6c0fa6f24f9a8cf45ea9cb981991'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:01.168568 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1393/2785] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:00 2022] DELETE /identity/v3/users/e73f6c0fa6f24f9a8cf45ea9cb981991 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:01.173221 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:01.173509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:01.173781 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:01.193557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:01.194831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:01.264124 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1393/2786] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:01.275391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:01.276827 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:01.337130 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d00f9c-11a4-4840-92eb-27a923a53766 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:01.342398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:01.343231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:01.343596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:01.344291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:01.358324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:01.360491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:01.372210 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1394/2787] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:01.945749 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:01.947256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:01.959811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fef268e-6734-47e5-80e4-3190a70a1e43 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:01.964875 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:01.965512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:01.965786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:01.966113 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] PATH_INFO: `/v3/projects/abe975f4c06944af8951be322f50858e` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:01.967394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=abe975f4c06944af8951be322f50858e)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:01.969520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:02.019979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'abe975f4c06944af8951be322f50858e', 'request_id': 'req-c76fb0e2-a73b-4f09-95d2-459003b96e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.028459 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'abe975f4c06944af8951be322f50858e', 'request_id': 'req-c76fb0e2-a73b-4f09-95d2-459003b96e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.035300 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'abe975f4c06944af8951be322f50858e', 'request_id': 'req-c76fb0e2-a73b-4f09-95d2-459003b96e65'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.042970 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:02.047118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] The token cache is being invalidate because project abe975f4c06944af8951be322f50858e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:02.047872 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.048718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.050418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1394/2788] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:01 2022] DELETE /identity/v3/projects/abe975f4c06944af8951be322f50858e => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:02.054861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:02.064885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00f5bf66-540f-4e46-89a2-7763b7abe740 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:02.125630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:02.126293 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:02.126559 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:02.126825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] PATH_INFO: `/v3/users/e4dac56cadc4463cb2c88db3ece9d2ad` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:02.127964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4dac56cadc4463cb2c88db3ece9d2ad)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:02.129861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:02.144666 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:02.188834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4dac56cadc4463cb2c88db3ece9d2ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.195711 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4dac56cadc4463cb2c88db3ece9d2ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.196368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4dac56cadc4463cb2c88db3ece9d2ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.196987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4dac56cadc4463cb2c88db3ece9d2ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.206181 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-26fe155d-743c-4af9-8414-e78400a0978e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4dac56cadc4463cb2c88db3ece9d2ad'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.224324 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1395/2789] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:02 2022] DELETE /identity/v3/users/e4dac56cadc4463cb2c88db3ece9d2ad => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:02.517699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:02.519388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:02.533695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c76fb0e2-a73b-4f09-95d2-459003b96e65 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:02.592230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:02.592861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:02.593129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:02.593397 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] PATH_INFO: `/v3/projects/5f5815fa4b2145f4a77525bdaed8b991` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:02.594539 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] RBAC: Authorizing `identity:delete_project(project_id=5f5815fa4b2145f4a77525bdaed8b991)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:02.596472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:02.648682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5f5815fa4b2145f4a77525bdaed8b991', 'request_id': 'req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.657444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5f5815fa4b2145f4a77525bdaed8b991', 'request_id': 'req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.664891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5f5815fa4b2145f4a77525bdaed8b991', 'request_id': 'req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.676284 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] The token cache is being invalidate because project 5f5815fa4b2145f4a77525bdaed8b991 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:02.676571 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.677268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-10d825a8-7d4d-4ed7-ba9c-588c580ebc3d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:02.679382 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1395/2790] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:02 2022] DELETE /identity/v3/projects/5f5815fa4b2145f4a77525bdaed8b991 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:02.684590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65231996-f02f-4660-a8e0-dfd811ec5eea None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:02.684870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65231996-f02f-4660-a8e0-dfd811ec5eea None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:02.685136 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65231996-f02f-4660-a8e0-dfd811ec5eea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:02.705964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-65231996-f02f-4660-a8e0-dfd811ec5eea None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:02.707202 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-65231996-f02f-4660-a8e0-dfd811ec5eea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:02.769895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1396/2791] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:02.778530 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd692bd-789c-47a2-8295-fb399dba7c5b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:02.778812 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd692bd-789c-47a2-8295-fb399dba7c5b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:02.779085 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd692bd-789c-47a2-8295-fb399dba7c5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:02.799426 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-abd692bd-789c-47a2-8295-fb399dba7c5b None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:02.800644 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-abd692bd-789c-47a2-8295-fb399dba7c5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:02.808968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1396/2792] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:02.813417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65231996-f02f-4660-a8e0-dfd811ec5eea None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:02.823840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65231996-f02f-4660-a8e0-dfd811ec5eea None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:02.828812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:02.829426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:02.829712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:02.829978 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:02.830659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:25:02.830961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:02.832743 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:02.845165 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1397/2793] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:25:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:25:02.849790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd692bd-789c-47a2-8295-fb399dba7c5b None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:02.859346 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd692bd-789c-47a2-8295-fb399dba7c5b None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:02.864964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:02.865646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:02.865908 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:02.866240 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:02.866868 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:02.868779 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:02.901402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1397/2794] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:02 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:25:02.905836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:02.916255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120fa153-eab2-4e79-93e9-4e37cae90a8e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:02.973332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:02.973933 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:02.974240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:02.974509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:02.975257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:02.977172 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.000677 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1398/2795] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:02 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:03.005100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.016669 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b152f5-880f-4c07-a3d6-96b1b6a5783b admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.021722 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.028770 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.029807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.030556 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.032197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.034936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.048092 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1398/2796] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:03 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:03.053826 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.065079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4a5695c-012c-4087-8ee1-a39f9c136905 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.070590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0966cd55-3e6f-4788-82e2-5d6144613352 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.071326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0966cd55-3e6f-4788-82e2-5d6144613352 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.071604 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0966cd55-3e6f-4788-82e2-5d6144613352 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.071870 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0966cd55-3e6f-4788-82e2-5d6144613352 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.072659 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0966cd55-3e6f-4788-82e2-5d6144613352 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.074309 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0966cd55-3e6f-4788-82e2-5d6144613352 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.076555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:03.094041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1399/2797] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:03 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:03.095618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.107439 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e74703-6cb5-4645-9d06-be15ce6caa90 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.112920 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.113567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.113838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.114185 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] PATH_INFO: `/v3/projects/3eca19b662fe47758e69e688d228869b/users/521933d62aa9440a9f6dfc3354b19fcd/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.129017 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3eca19b662fe47758e69e688d228869b, user_id=521933d62aa9440a9f6dfc3354b19fcd, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.131688 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.147724 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1399/2798] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:03 2022] PUT /identity/v3/projects/3eca19b662fe47758e69e688d228869b/users/521933d62aa9440a9f6dfc3354b19fcd/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:03.159238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0137b85d-c7c5-4e77-bfa7-809d10fd8bec None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.159467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0137b85d-c7c5-4e77-bfa7-809d10fd8bec None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.159738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0137b85d-c7c5-4e77-bfa7-809d10fd8bec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.189075 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0137b85d-c7c5-4e77-bfa7-809d10fd8bec None None] MFA Rules not processed for user `521933d62aa9440a9f6dfc3354b19fcd`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:03.190285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0137b85d-c7c5-4e77-bfa7-809d10fd8bec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:03.264292 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1400/2799] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:03 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:03.265620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.296790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:03.330763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6f9d60c-f62b-4f43-8746-30be1a4bf7d1 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.335801 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.336518 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.336783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.337066 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.337686 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.339682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.368898 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1400/2800] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:03 2022] POST /identity/v3/projects => generated 385 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:03.373197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0137b85d-c7c5-4e77-bfa7-809d10fd8bec None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.384963 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0137b85d-c7c5-4e77-bfa7-809d10fd8bec None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.444284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.444920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.445269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.445607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.446282 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.448195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.472435 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1401/2801] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:03 2022] POST /identity/v3/users => generated 293 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:25:03.476804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.488170 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39251c5-a2fc-4724-82ab-ea24f15c780c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.493263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.493855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.494293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.494563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.495197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.496806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.508494 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1401/2802] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:03.513841 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.523512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc522028-fb20-4287-bfae-37ce0d52644c admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.528684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14fbd1a6-27be-4c00-98aa-e49cf6ca0a7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.529273 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fbd1a6-27be-4c00-98aa-e49cf6ca0a7c admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.529563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fbd1a6-27be-4c00-98aa-e49cf6ca0a7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.529830 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14fbd1a6-27be-4c00-98aa-e49cf6ca0a7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.530523 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fbd1a6-27be-4c00-98aa-e49cf6ca0a7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.532102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14fbd1a6-27be-4c00-98aa-e49cf6ca0a7c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.544378 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1402/2803] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:03 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:03.548861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.559685 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8975d8a-008e-4ed6-98e0-33c71ef5e2a5 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.564618 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.565263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.565534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.565795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] PATH_INFO: `/v3/projects/22e84a896c6b4809a6896e5df56c1d59/users/984caea3c9514eb88fcfeab1b91ada53/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.580979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] RBAC: Authorizing `identity:create_grant(project_id=22e84a896c6b4809a6896e5df56c1d59, user_id=984caea3c9514eb88fcfeab1b91ada53, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.583648 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.598904 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1402/2804] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:03 2022] PUT /identity/v3/projects/22e84a896c6b4809a6896e5df56c1d59/users/984caea3c9514eb88fcfeab1b91ada53/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:03.609381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eefe49c-4e79-455c-bbf2-6b2615203182 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.609661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eefe49c-4e79-455c-bbf2-6b2615203182 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.609926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eefe49c-4e79-455c-bbf2-6b2615203182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.637565 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8eefe49c-4e79-455c-bbf2-6b2615203182 None None] MFA Rules not processed for user `984caea3c9514eb88fcfeab1b91ada53`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:03.638780 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8eefe49c-4e79-455c-bbf2-6b2615203182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:03.695331 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:03.707505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1403/2805] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:03 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:03.713246 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.778099 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2d14fc-42a5-4485-bdd6-cb9914bf1093 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.783197 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.783797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.784071 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.784343 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.784956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.786903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.815895 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1403/2806] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:03 2022] POST /identity/v3/projects => generated 385 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:03.820379 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eefe49c-4e79-455c-bbf2-6b2615203182 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.831275 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eefe49c-4e79-455c-bbf2-6b2615203182 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.889602 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.890230 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.890504 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.890765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.891353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.893318 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.917683 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1404/2807] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:03 2022] POST /identity/v3/users => generated 299 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:03.924086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.937334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424dcddf-3e35-4d66-9561-ad00285d33eb admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.944018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.944294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.944509 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.944732 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.945444 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.947031 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:03.958373 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1404/2808] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:03 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:03.963432 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:03.973579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41db93d0-60c4-4b11-a3bf-90497fc6b0d5 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:03.979439 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:03.980112 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:03.980382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:03.980648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] PATH_INFO: `/v3/projects/c582e218270f4adbb70ea7865361d3ad/users/c1da7bc07c7145cda579f2b300ca0dd0/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:03.995601 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c582e218270f4adbb70ea7865361d3ad, user_id=c1da7bc07c7145cda579f2b300ca0dd0, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:03.998500 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:04.013339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1405/2809] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:03 2022] PUT /identity/v3/projects/c582e218270f4adbb70ea7865361d3ad/users/c1da7bc07c7145cda579f2b300ca0dd0/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:25:04.018345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:04.033463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aefa04bc-c04d-4311-b215-38c0e6afa861 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:04.087839 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:04.103188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c04e7a8-d7c2-4c55-96b9-55c32528754d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:04.103795 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c04e7a8-d7c2-4c55-96b9-55c32528754d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:04.104072 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c04e7a8-d7c2-4c55-96b9-55c32528754d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:04.104341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c04e7a8-d7c2-4c55-96b9-55c32528754d admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:04.104984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c04e7a8-d7c2-4c55-96b9-55c32528754d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:04.106671 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c04e7a8-d7c2-4c55-96b9-55c32528754d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:04.116681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1405/2810] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:04 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:25:04.121127 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:04.131791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee99778-e5cc-47c3-933f-9a23bc064830 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:04.136995 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:04.137656 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:04.137925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:04.138238 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] PATH_INFO: `/v3/projects/c582e218270f4adbb70ea7865361d3ad/users/c1da7bc07c7145cda579f2b300ca0dd0/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:04.140526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c582e218270f4adbb70ea7865361d3ad, user_id=c1da7bc07c7145cda579f2b300ca0dd0, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:04.143175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:04.154074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1406/2811] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:04 2022] PUT /identity/v3/projects/c582e218270f4adbb70ea7865361d3ad/users/c1da7bc07c7145cda579f2b300ca0dd0/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:04.164512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39773b7f-37e0-4909-9b50-dff6107c45a7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:04.164800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39773b7f-37e0-4909-9b50-dff6107c45a7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:04.165061 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39773b7f-37e0-4909-9b50-dff6107c45a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:04.193574 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-39773b7f-37e0-4909-9b50-dff6107c45a7 None None] MFA Rules not processed for user `c1da7bc07c7145cda579f2b300ca0dd0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:04.195028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-39773b7f-37e0-4909-9b50-dff6107c45a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:04.270691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1406/2812] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:04 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:04.283252 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:04.285918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:04.350267 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7fcc14-c64e-4efc-a686-b2d346a9bdd9 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:04.357362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:04.358353 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:04.358765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:04.359144 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:04.422689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:04.424166 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:04.433441 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1407/2813] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:04 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:06.388028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b51b67e-82c7-4f9e-925c-6baf3ec70b53 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:06.388570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b51b67e-82c7-4f9e-925c-6baf3ec70b53 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:06.388570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b51b67e-82c7-4f9e-925c-6baf3ec70b53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:06.409289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8b51b67e-82c7-4f9e-925c-6baf3ec70b53 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:06.410693 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8b51b67e-82c7-4f9e-925c-6baf3ec70b53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:06.467698 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1407/2814] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:25:06 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:06.520555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:06.521987 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:06.527755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:06.597270 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c4fef0-5487-46d4-b47b-d61789f1fa13 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:06.603506 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:06.604110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:06.604388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:06.604646 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:06.613640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:06.615261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:06.623813 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1408/2815] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:25:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:25:07.075672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b51b67e-82c7-4f9e-925c-6baf3ec70b53 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:07.090492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b51b67e-82c7-4f9e-925c-6baf3ec70b53 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:07.095196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:07.096019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:07.096552 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:07.096741 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:07.174759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:07.176379 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:07.186833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1408/2816] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:07 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:25:07.834697 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:07.845382 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9849de2e-3e2f-4a72-ac61-ac4d2e6a204d service swift] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:07.849670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e12c21-befd-4fe1-8b20-9a15efe0af79 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:07.850368 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e12c21-befd-4fe1-8b20-9a15efe0af79 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:07.850734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e12c21-befd-4fe1-8b20-9a15efe0af79 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:07.851071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e12c21-befd-4fe1-8b20-9a15efe0af79 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:07.863921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e12c21-befd-4fe1-8b20-9a15efe0af79 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:07.865360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e12c21-befd-4fe1-8b20-9a15efe0af79 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:07.875644 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1409/2817] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:07 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:13.068287 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:13.087640 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=100 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:13.140256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc76ba9-146c-4796-bbe5-79af31b9f409 service cinder] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:13.145018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:13.145831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:13.146164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:13.146438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] PATH_INFO: `/v3/users/521933d62aa9440a9f6dfc3354b19fcd` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:13.147689 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=521933d62aa9440a9f6dfc3354b19fcd)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:13.149639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:13.216483 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '521933d62aa9440a9f6dfc3354b19fcd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.227172 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '521933d62aa9440a9f6dfc3354b19fcd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.228111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '521933d62aa9440a9f6dfc3354b19fcd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.228784 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '521933d62aa9440a9f6dfc3354b19fcd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.238665 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '521933d62aa9440a9f6dfc3354b19fcd'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.261202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1409/2818] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:13 2022] DELETE /identity/v3/users/521933d62aa9440a9f6dfc3354b19fcd => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:13.266082 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:13.266589 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:13.266816 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:13.287044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:13.287964 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:13.360514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1410/2819] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:13.367994 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:13.370074 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:13.440943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd6e4cd-4a27-4831-9a63-311f964aa1b9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:13.445454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:13.446186 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:13.446523 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:13.446855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:13.456869 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:13.458674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:13.469281 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1410/2820] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:13.802529 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:13.804263 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:13.817071 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f81320f8-deba-4105-b5ea-87d813332fc9 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:13.823822 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:13.824834 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:13.825438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:13.825927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] PATH_INFO: `/v3/projects/3eca19b662fe47758e69e688d228869b` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:13.827680 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3eca19b662fe47758e69e688d228869b)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:13.830888 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:13.890867 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3eca19b662fe47758e69e688d228869b', 'request_id': 'req-2df1412f-914a-41e8-9782-8e2b976fd3a7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.904307 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3eca19b662fe47758e69e688d228869b', 'request_id': 'req-2df1412f-914a-41e8-9782-8e2b976fd3a7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.912748 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3eca19b662fe47758e69e688d228869b', 'request_id': 'req-2df1412f-914a-41e8-9782-8e2b976fd3a7'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.925175 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] The token cache is being invalidate because project 3eca19b662fe47758e69e688d228869b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:13.925493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.926515 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:13.929442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1411/2821] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:13 2022] DELETE /identity/v3/projects/3eca19b662fe47758e69e688d228869b => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:13.933230 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:13.943453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-407c273c-bc2c-45e9-a935-2c5a94a5a96b service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:13.980531 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:14.009012 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:14.009807 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:14.010231 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:14.010601 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] PATH_INFO: `/v3/users/984caea3c9514eb88fcfeab1b91ada53` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:14.012199 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] RBAC: Authorizing `identity:delete_user(user_id=984caea3c9514eb88fcfeab1b91ada53)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:14.014326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:14.081188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '984caea3c9514eb88fcfeab1b91ada53'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.090591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '984caea3c9514eb88fcfeab1b91ada53'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.091296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '984caea3c9514eb88fcfeab1b91ada53'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.091889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '984caea3c9514eb88fcfeab1b91ada53'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.103081 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '984caea3c9514eb88fcfeab1b91ada53'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.128520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1411/2822] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:13 2022] DELETE /identity/v3/users/984caea3c9514eb88fcfeab1b91ada53 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:14.463871 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:14.465629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:14.474810 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:14.496116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df1412f-914a-41e8-9782-8e2b976fd3a7 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:14.575284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:14.575942 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:14.576326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:14.576594 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] PATH_INFO: `/v3/projects/22e84a896c6b4809a6896e5df56c1d59` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:14.577708 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] RBAC: Authorizing `identity:delete_project(project_id=22e84a896c6b4809a6896e5df56c1d59)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:14.579655 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:14.645447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '22e84a896c6b4809a6896e5df56c1d59', 'request_id': 'req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.655784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '22e84a896c6b4809a6896e5df56c1d59', 'request_id': 'req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.663981 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '22e84a896c6b4809a6896e5df56c1d59', 'request_id': 'req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.676490 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] The token cache is being invalidate because project 22e84a896c6b4809a6896e5df56c1d59 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:14.676799 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.677563 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.679467 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1412/2823] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:14 2022] DELETE /identity/v3/projects/22e84a896c6b4809a6896e5df56c1d59 => generated 0 bytes in 216 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:14.683596 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:14.685370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:14.709107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cded75c-6d0c-48e7-94b9-ea6e2dc11205 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:14.786613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:14.787563 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:14.788091 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:14.788630 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] PATH_INFO: `/v3/users/c1da7bc07c7145cda579f2b300ca0dd0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:14.790299 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] RBAC: Authorizing `identity:delete_user(user_id=c1da7bc07c7145cda579f2b300ca0dd0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:14.792854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:14.855862 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c1da7bc07c7145cda579f2b300ca0dd0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.863210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c1da7bc07c7145cda579f2b300ca0dd0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.864111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c1da7bc07c7145cda579f2b300ca0dd0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.864921 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c1da7bc07c7145cda579f2b300ca0dd0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.867670 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:14.875916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c1da7bc07c7145cda579f2b300ca0dd0'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:14.903330 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1412/2824] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:14 2022] DELETE /identity/v3/users/c1da7bc07c7145cda579f2b300ca0dd0 => generated 0 bytes in 220 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:15.224159 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:15.225970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:15.241193 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fa3ba5-69a8-4428-aae1-449f5dbeb75e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:15.314652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:15.315648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:15.315925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:15.316197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] PATH_INFO: `/v3/projects/c582e218270f4adbb70ea7865361d3ad` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:15.317660 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c582e218270f4adbb70ea7865361d3ad)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:15.319661 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:15.364986 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:15.380762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c582e218270f4adbb70ea7865361d3ad', 'request_id': 'req-82519bb5-ebde-4663-aa7a-76c1ae2457c1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:15.392484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c582e218270f4adbb70ea7865361d3ad', 'request_id': 'req-82519bb5-ebde-4663-aa7a-76c1ae2457c1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:15.400438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c582e218270f4adbb70ea7865361d3ad', 'request_id': 'req-82519bb5-ebde-4663-aa7a-76c1ae2457c1'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:15.415470 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] The token cache is being invalidate because project c582e218270f4adbb70ea7865361d3ad was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:15.415849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:15.416578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-82519bb5-ebde-4663-aa7a-76c1ae2457c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:15.418170 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1413/2825] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:15 2022] DELETE /identity/v3/projects/c582e218270f4adbb70ea7865361d3ad => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:18.564989 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:18.566619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:18.580871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59dfa098-e6f1-44a8-9b65-2ea42da1026e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:18.654465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:18.655316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:18.655595 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:18.655867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] PATH_INFO: `/v3/users/99a362c0101540e997c3d39de297f502` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:18.657144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=99a362c0101540e997c3d39de297f502)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:18.659451 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:18.712570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '99a362c0101540e997c3d39de297f502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:18.725680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '99a362c0101540e997c3d39de297f502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:18.726506 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '99a362c0101540e997c3d39de297f502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:18.727233 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '99a362c0101540e997c3d39de297f502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:18.739307 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '99a362c0101540e997c3d39de297f502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:18.769391 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1413/2826] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:18 2022] DELETE /identity/v3/users/99a362c0101540e997c3d39de297f502 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:18.774078 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:18.774736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:18.774736 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:18.797186 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:18.798355 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:18.868635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1414/2827] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:18.879943 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:18.881798 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:18.954668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7495a86e-804d-420d-8e8c-8e30faa807c4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:18.958345 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e20f2f2f-3b5d-4cfb-ba01-a493c21fc8db service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:18.959661 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f2f2f-3b5d-4cfb-ba01-a493c21fc8db service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:18.959937 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f2f2f-3b5d-4cfb-ba01-a493c21fc8db service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:18.960206 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e20f2f2f-3b5d-4cfb-ba01-a493c21fc8db service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:18.972058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20f2f2f-3b5d-4cfb-ba01-a493c21fc8db service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:18.973777 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e20f2f2f-3b5d-4cfb-ba01-a493c21fc8db service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:18.978187 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:18.985479 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1414/2828] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 29 02:25:19.316044 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:19.317730 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:19.339129 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddf4916e-ddfc-45d8-8daa-487386998c45 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:19.352828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:19.353846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.354359 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.354825 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] PATH_INFO: `/v3/projects/3968c76fd4464c4dab0977f88694b3e1` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.356360 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3968c76fd4464c4dab0977f88694b3e1)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:19.360096 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:19.424939 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3968c76fd4464c4dab0977f88694b3e1', 'request_id': 'req-c19f2985-a388-43c1-b6cb-1310a4b11e3c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:19.436849 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3968c76fd4464c4dab0977f88694b3e1', 'request_id': 'req-c19f2985-a388-43c1-b6cb-1310a4b11e3c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:19.444608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3968c76fd4464c4dab0977f88694b3e1', 'request_id': 'req-c19f2985-a388-43c1-b6cb-1310a4b11e3c'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:19.458681 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] The token cache is being invalidate because project 3968c76fd4464c4dab0977f88694b3e1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:19.459065 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:19.460041 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-c19f2985-a388-43c1-b6cb-1310a4b11e3c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:19.461764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1415/2829] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:19 2022] DELETE /identity/v3/projects/3968c76fd4464c4dab0977f88694b3e1 => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:19.467100 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37d1c6-316b-4c8b-8303-9af341b6544c None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.467501 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37d1c6-316b-4c8b-8303-9af341b6544c None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.468041 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb37d1c6-316b-4c8b-8303-9af341b6544c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.488636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-bb37d1c6-316b-4c8b-8303-9af341b6544c None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:19.490115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-bb37d1c6-316b-4c8b-8303-9af341b6544c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:19.573347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1415/2830] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:19.585303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000a8664-4118-4eba-b266-fcc5e0ba58dc None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.585625 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000a8664-4118-4eba-b266-fcc5e0ba58dc None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.585977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000a8664-4118-4eba-b266-fcc5e0ba58dc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.613468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-000a8664-4118-4eba-b266-fcc5e0ba58dc None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:19.614861 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-000a8664-4118-4eba-b266-fcc5e0ba58dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:19.626494 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1416/2831] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:25:19.627753 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37d1c6-316b-4c8b-8303-9af341b6544c None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:19.640294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb37d1c6-316b-4c8b-8303-9af341b6544c None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:19.646029 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:19.646756 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.647279 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.647799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.648736 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:25:19.649249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:19.651555 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:19.666972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1416/2832] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:25:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:25:19.671352 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000a8664-4118-4eba-b266-fcc5e0ba58dc None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:19.682745 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000a8664-4118-4eba-b266-fcc5e0ba58dc None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:19.688980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:19.689699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.690059 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.690394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.691143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:19.693423 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:19.735443 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1417/2833] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:19 2022] POST /identity/v3/projects => generated 383 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:19.736649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:19.747590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce8543ac-4ad4-4c60-96f6-3a3728ba04b7 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:19.820545 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:19.821158 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.821432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.821700 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.822378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:19.824297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:19.855633 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1417/2834] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:19 2022] POST /identity/v3/users => generated 292 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:19.857063 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:19.870137 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67f7001-8f2c-412c-a125-49167c79c07e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:19.877143 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:19.878689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.878943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.879211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.880406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:19.882539 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:19.894844 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1418/2835] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:19 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:19.899075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:19.912745 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a43550f-40c3-4193-a353-b7ea63f0701e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:19.919691 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071face9-566d-493e-bcd4-6aa635d44504 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:19.920763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071face9-566d-493e-bcd4-6aa635d44504 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.921019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071face9-566d-493e-bcd4-6aa635d44504 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.921399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071face9-566d-493e-bcd4-6aa635d44504 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.922250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071face9-566d-493e-bcd4-6aa635d44504 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:19.924317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071face9-566d-493e-bcd4-6aa635d44504 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:19.937194 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1418/2836] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:19 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:19.941459 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:19.953704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4964a4e2-660a-4db5-b4a9-f3b20597f3b8 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:19.961114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:19.962114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:19.962729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:19.963133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] PATH_INFO: `/v3/projects/5c2d79eb0f9f48b5aa72e8cf200bd8af/users/86a52ab6060244b3b7b3a88bbdc782b1/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:19.966246 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:19.984936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5c2d79eb0f9f48b5aa72e8cf200bd8af, user_id=86a52ab6060244b3b7b3a88bbdc782b1, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:19.988037 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:20.004519 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1419/2837] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:19 2022] PUT /identity/v3/projects/5c2d79eb0f9f48b5aa72e8cf200bd8af/users/86a52ab6060244b3b7b3a88bbdc782b1/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:20.016086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630b27f-cb66-4052-8c0d-781f82944c8b None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:20.016445 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630b27f-cb66-4052-8c0d-781f82944c8b None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:20.016718 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7630b27f-cb66-4052-8c0d-781f82944c8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:20.051010 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-7630b27f-cb66-4052-8c0d-781f82944c8b None None] MFA Rules not processed for user `86a52ab6060244b3b7b3a88bbdc782b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:20.054433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-7630b27f-cb66-4052-8c0d-781f82944c8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:20.117939 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:20.131695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1419/2838] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:20 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:25:20.143973 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:20.146240 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:20.211576 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3be36c9-779c-4977-89a4-7de509a47760 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:20.215451 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:20.216194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:20.216417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:20.216670 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:20.226610 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:20.228512 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:20.240101 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1420/2839] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:20 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:27.836706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4c63d-6c05-4e61-99e6-6f3f8c9a6034 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:27.837204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4c63d-6c05-4e61-99e6-6f3f8c9a6034 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:27.837204 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dd4c63d-6c05-4e61-99e6-6f3f8c9a6034 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:27.860797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2dd4c63d-6c05-4e61-99e6-6f3f8c9a6034 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:27.862028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2dd4c63d-6c05-4e61-99e6-6f3f8c9a6034 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:27.919590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1420/2840] 10.222.0.61 () {58 vars in 967 bytes} [Thu Sep 29 02:25:27 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:27.932652 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:27.934428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:28.011725 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fedb572-2ae1-4550-81c3-6fc7b3135d82 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:28.016689 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:28.017285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:28.017554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:28.017824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:28.025894 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:28.027493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:28.035627 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1421/2841] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:25:27 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:25:29.950841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd4c63d-6c05-4e61-99e6-6f3f8c9a6034 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.015107 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dd4c63d-6c05-4e61-99e6-6f3f8c9a6034 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.019761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.020423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.020679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.020971 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] PATH_INFO: `/v3/users/86a52ab6060244b3b7b3a88bbdc782b1` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.022102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] RBAC: Authorizing `identity:delete_user(user_id=86a52ab6060244b3b7b3a88bbdc782b1)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.024198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.089878 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '86a52ab6060244b3b7b3a88bbdc782b1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.098347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.099882 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.101455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '86a52ab6060244b3b7b3a88bbdc782b1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.102276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '86a52ab6060244b3b7b3a88bbdc782b1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.102871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '86a52ab6060244b3b7b3a88bbdc782b1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.116206 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94dfd377-f9ad-4500-9467-5ffaadc73c19 service swift] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.116525 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-d2af9c30-4343-445f-90f1-c66d9687e693 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '86a52ab6060244b3b7b3a88bbdc782b1'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.121349 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.121984 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.122288 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.122558 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] PATH_INFO: `/v3/users/d83a16ba92834ab6a23a4b4634711178` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.123761 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] RBAC: Authorizing `identity:delete_user(user_id=d83a16ba92834ab6a23a4b4634711178)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.125678 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.136210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1421/2842] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:29 2022] DELETE /identity/v3/users/86a52ab6060244b3b7b3a88bbdc782b1 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:30.140861 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.141220 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.141590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.161758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:30.163119 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.175740 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:30.176033 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd83a16ba92834ab6a23a4b4634711178'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.184241 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd83a16ba92834ab6a23a4b4634711178'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.184824 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd83a16ba92834ab6a23a4b4634711178'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.185417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd83a16ba92834ab6a23a4b4634711178'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.194370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd83a16ba92834ab6a23a4b4634711178'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.201219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:30.216447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1422/2843] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:30 2022] DELETE /identity/v3/users/d83a16ba92834ab6a23a4b4634711178 => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:25:30.236892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1422/2844] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:25:30.251949 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.254574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.315864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9097c003-1068-420a-a2f3-0f07bf8482ab admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.319617 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5bcbdc9-2fff-4712-88fa-95fdfdb61bc0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.320224 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bcbdc9-2fff-4712-88fa-95fdfdb61bc0 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.320493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bcbdc9-2fff-4712-88fa-95fdfdb61bc0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.320762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5bcbdc9-2fff-4712-88fa-95fdfdb61bc0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.382023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bcbdc9-2fff-4712-88fa-95fdfdb61bc0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.383690 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5bcbdc9-2fff-4712-88fa-95fdfdb61bc0 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.394391 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1423/2845] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:30.395276 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.396771 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.409237 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fd536cd-e059-46b8-8327-97dc239fb567 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.416584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.417322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.417660 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.417991 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] PATH_INFO: `/v3/projects/04420159627d4beeb44bb038e633a96d` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.419269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] RBAC: Authorizing `identity:delete_project(project_id=04420159627d4beeb44bb038e633a96d)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.421561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.476611 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '04420159627d4beeb44bb038e633a96d', 'request_id': 'req-fb337288-e45b-47e7-b222-99a3d96ec34b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.485590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '04420159627d4beeb44bb038e633a96d', 'request_id': 'req-fb337288-e45b-47e7-b222-99a3d96ec34b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.493090 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '04420159627d4beeb44bb038e633a96d', 'request_id': 'req-fb337288-e45b-47e7-b222-99a3d96ec34b'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.504202 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] The token cache is being invalidate because project 04420159627d4beeb44bb038e633a96d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:30.504613 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.505616 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-fb337288-e45b-47e7-b222-99a3d96ec34b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.507174 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1423/2846] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:30 2022] DELETE /identity/v3/projects/04420159627d4beeb44bb038e633a96d => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:25:30.513358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c95b7-9d59-48de-88df-e70a914e4aff None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.513704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c95b7-9d59-48de-88df-e70a914e4aff None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.514073 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450c95b7-9d59-48de-88df-e70a914e4aff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.541468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-450c95b7-9d59-48de-88df-e70a914e4aff None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:30.542526 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-450c95b7-9d59-48de-88df-e70a914e4aff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.608833 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1424/2847] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:30.618097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.618381 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.618656 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.641383 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:30.642649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.651375 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1424/2848] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:30.655509 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450c95b7-9d59-48de-88df-e70a914e4aff None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.665890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450c95b7-9d59-48de-88df-e70a914e4aff None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.671197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.672000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.672223 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.672466 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.673132 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:25:30.673431 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.675555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.679968 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.681465 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.689211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1425/2849] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:25:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 29 02:25:30.693590 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.694620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4a76767-95d7-4b54-868c-22f840b1f265 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.699473 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.700104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.700392 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.700804 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] PATH_INFO: `/v3/projects/5c2d79eb0f9f48b5aa72e8cf200bd8af` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.701929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5c2d79eb0f9f48b5aa72e8cf200bd8af)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.703884 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.704535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33bc4699-f270-47dc-bc97-889fa837b8e1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.710035 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.710643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.710917 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.711189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.711817 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.713770 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.745428 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1426/2850] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/projects => generated 371 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:25:30.749549 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.753935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=98,keystone:DELETE=1,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:30.755478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5c2d79eb0f9f48b5aa72e8cf200bd8af', 'request_id': 'req-6b144baa-b769-4e47-b6ef-77eecb7b3d79'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.766093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5c2d79eb0f9f48b5aa72e8cf200bd8af', 'request_id': 'req-6b144baa-b769-4e47-b6ef-77eecb7b3d79'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.772844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5c2d79eb0f9f48b5aa72e8cf200bd8af', 'request_id': 'req-6b144baa-b769-4e47-b6ef-77eecb7b3d79'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.783311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] The token cache is being invalidate because project 5c2d79eb0f9f48b5aa72e8cf200bd8af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:30.783600 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.784247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-6b144baa-b769-4e47-b6ef-77eecb7b3d79 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:30.786060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1425/2851] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:30 2022] DELETE /identity/v3/projects/5c2d79eb0f9f48b5aa72e8cf200bd8af => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:30.790481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.790759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.791036 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.809949 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:30.811164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.817177 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966760e4-2346-4007-b1f0-faba14ac1f3f admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.819262 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1426/2852] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:25:30.821943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4108245f-07ab-4809-be6a-5105c3792b14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.822575 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108245f-07ab-4809-be6a-5105c3792b14 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.822846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108245f-07ab-4809-be6a-5105c3792b14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.823114 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108245f-07ab-4809-be6a-5105c3792b14 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.823741 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4108245f-07ab-4809-be6a-5105c3792b14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.825658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4108245f-07ab-4809-be6a-5105c3792b14 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.852461 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1427/2853] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/users => generated 286 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:25:30.856790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.858362 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.871130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a4685e-8399-4afc-b9d1-63e6483159a5 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.876135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.876760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.877032 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.877296 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.877942 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.879679 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.886622 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:30.887585 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd50c14-7968-4aa3-b69c-b02f63e4a2e0 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.887887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd50c14-7968-4aa3-b69c-b02f63e4a2e0 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.888199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dd50c14-7968-4aa3-b69c-b02f63e4a2e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.892078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1427/2854] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:30 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:30.896115 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.906372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9d8a78-2c12-41e3-81c6-b148ca1ce6e4 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.909348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-3dd50c14-7968-4aa3-b69c-b02f63e4a2e0 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:30.910837 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-3dd50c14-7968-4aa3-b69c-b02f63e4a2e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:30.910977 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.911490 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.911760 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.912023 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.912649 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.914269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.919948 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1428/2855] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:25:30.924185 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd50c14-7968-4aa3-b69c-b02f63e4a2e0 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.924621 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1428/2856] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:30 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:30.928636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.934334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dd50c14-7968-4aa3-b69c-b02f63e4a2e0 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.938360 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cdc55f2-87fe-4ece-a2dc-4828814271e6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.939758 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.940370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.940645 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.941076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.941715 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:25:30.942030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.943407 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d617cbe-c6a9-436d-bd85-1eb8dcb849ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.943640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.944038 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d617cbe-c6a9-436d-bd85-1eb8dcb849ba admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.944309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d617cbe-c6a9-436d-bd85-1eb8dcb849ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.944575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d617cbe-c6a9-436d-bd85-1eb8dcb849ba admin admin] PATH_INFO: `/v3/projects/a2f2d778c35d4a90a28aeef64fefa8fb/users/6ce84f7e891242c393d0ac5e97c8a9e2/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.955612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1429/2857] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:25:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:25:30.959438 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d617cbe-c6a9-436d-bd85-1eb8dcb849ba admin admin] RBAC: Authorizing `identity:create_grant(project_id=a2f2d778c35d4a90a28aeef64fefa8fb, user_id=6ce84f7e891242c393d0ac5e97c8a9e2, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.959921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:30.962096 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d617cbe-c6a9-436d-bd85-1eb8dcb849ba admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.970058 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2412a3a-8a53-4445-9e5a-aefc0f364c37 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:30.975604 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:30.976237 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.976505 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.976767 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:30.977384 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:30.978792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1429/2858] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:30 2022] PUT /identity/v3/projects/a2f2d778c35d4a90a28aeef64fefa8fb/users/6ce84f7e891242c393d0ac5e97c8a9e2/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:25:30.982493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:30.983963 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:30.984269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:30.984537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.003858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:31.005461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:31.010313 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1430/2859] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/projects => generated 377 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:25:31.014438 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:31.069935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1430/2860] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:31.079390 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:31.081274 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:31.085932 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe7d2d12-4cb5-4a80-b39f-8035a8131000 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:31.091094 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:31.091721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:31.091992 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:31.092257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.092881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:31.094842 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:31.117480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1431/2861] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:31 2022] POST /identity/v3/users => generated 289 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:25:31.121635 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:31.131864 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b0b8bf-15c3-4cde-8c9f-1addb7f31d04 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:31.136493 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:31.137080 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:31.137348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:31.137667 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.138336 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:31.139916 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:31.141744 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea97c44-1f73-4352-b83f-129fca424285 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:31.149968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1432/2862] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:31 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:31.153910 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:31.164057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6bca3ec-d12e-401d-8735-b62acf9db262 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:31.168895 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:31.169587 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:31.169868 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:31.170200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.170831 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:31.172413 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:31.183189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1433/2863] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:31 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:25:31.187394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:31.196128 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28e66bd2-e964-410f-8251-defaefac5d15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:31.196782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e66bd2-e964-410f-8251-defaefac5d15 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:31.197058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e66bd2-e964-410f-8251-defaefac5d15 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:31.197164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ceeaf27-a2a7-4604-8af4-9b5810e958bf admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:31.197322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28e66bd2-e964-410f-8251-defaefac5d15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.202212 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:31.202840 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:31.203110 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:31.203374 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] PATH_INFO: `/v3/projects/c25e1b1ada2349a896c46d992833023e/users/8b1e13586ae54909adce04ca214f1365/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.206535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e66bd2-e964-410f-8251-defaefac5d15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:31.207972 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28e66bd2-e964-410f-8251-defaefac5d15 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:31.217614 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1431/2864] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:31.218330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c25e1b1ada2349a896c46d992833023e, user_id=8b1e13586ae54909adce04ca214f1365, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:31.221039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:31.223348 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:31.236030 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1434/2865] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:31 2022] PUT /identity/v3/projects/c25e1b1ada2349a896c46d992833023e/users/8b1e13586ae54909adce04ca214f1365/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 29 02:25:31.246935 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71868712-5c20-479d-b7d0-44a213669834 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:31.247214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71868712-5c20-479d-b7d0-44a213669834 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:31.247561 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71868712-5c20-479d-b7d0-44a213669834 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.277750 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-71868712-5c20-479d-b7d0-44a213669834 None None] MFA Rules not processed for user `8b1e13586ae54909adce04ca214f1365`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:31.279111 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-71868712-5c20-479d-b7d0-44a213669834 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:31.292754 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:31.350802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1432/2866] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:31.363591 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:31.365407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:31.430674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45585618-2b8e-4ca8-a0ce-97cd07c1edb1 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:31.434340 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:31.434946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:31.435216 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:31.435481 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:31.444398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:31.445836 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:31.454751 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1435/2867] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:35.246699 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5bcca-e74f-4151-9006-3706d0a16215 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:35.247268 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5bcca-e74f-4151-9006-3706d0a16215 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:35.247370 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0d5bcca-e74f-4151-9006-3706d0a16215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:35.277401 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-d0d5bcca-e74f-4151-9006-3706d0a16215 None None] MFA Rules not processed for user `6ce84f7e891242c393d0ac5e97c8a9e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:35.278672 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d0d5bcca-e74f-4151-9006-3706d0a16215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:35.346083 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1433/2868] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:35 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:35.351020 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:35.415079 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d166b1af-52fb-4546-b01c-c02aa283a8ff service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:35.419859 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:35.420543 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:35.420876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:35.421207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:35.421918 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:35.423982 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:35.458669 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1436/2869] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:35 2022] POST /identity/v3/projects => generated 373 bytes in 108 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:35.462403 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5bcca-e74f-4151-9006-3706d0a16215 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:35.473520 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0d5bcca-e74f-4151-9006-3706d0a16215 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:35.544132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:35.544730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:35.545009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:35.545436 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:35.546253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:35.548236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:35.577133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1434/2870] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:35 2022] POST /identity/v3/users => generated 293 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:35.578141 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:35.588929 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81f31823-4d93-47b1-ab2b-f78521605577 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:35.594162 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:35.594843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:35.595322 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:35.595640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:35.596339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:35.598026 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:35.609943 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1437/2871] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:25:35.614249 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:35.625288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d892b25c-1de0-4e9c-ba60-c4f28ea87266 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:35.631810 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:35.632564 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:35.632841 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:35.633110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] PATH_INFO: `/v3/projects/6b66385ef54a4c58af4bb356e1550354/users/26cf87bd28c64f4c9310d680510a80d4/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:35.649792 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b66385ef54a4c58af4bb356e1550354, user_id=26cf87bd28c64f4c9310d680510a80d4, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:35.652789 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:35.669289 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1435/2872] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:35 2022] PUT /identity/v3/projects/6b66385ef54a4c58af4bb356e1550354/users/26cf87bd28c64f4c9310d680510a80d4/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:35.673903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:35.685255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32fe0365-5c44-4b51-aff0-2394b17301f4 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:35.744853 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3532bbca-103d-4bbc-9366-016343812955 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:35.745426 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3532bbca-103d-4bbc-9366-016343812955 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:35.745732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3532bbca-103d-4bbc-9366-016343812955 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:35.746014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3532bbca-103d-4bbc-9366-016343812955 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:35.746698 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3532bbca-103d-4bbc-9366-016343812955 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:35.748290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3532bbca-103d-4bbc-9366-016343812955 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:35.759092 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1438/2873] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:35 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:25:35.763495 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:35.774135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc350e-3e55-4915-9971-d7c650434c92 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:35.778948 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:35.779635 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:35.779999 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:35.780333 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] PATH_INFO: `/v3/projects/6b66385ef54a4c58af4bb356e1550354/users/26cf87bd28c64f4c9310d680510a80d4/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:35.782394 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] RBAC: Authorizing `identity:create_grant(project_id=6b66385ef54a4c58af4bb356e1550354, user_id=26cf87bd28c64f4c9310d680510a80d4, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:35.785238 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:35.788659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:35.798198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1436/2874] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:35 2022] PUT /identity/v3/projects/6b66385ef54a4c58af4bb356e1550354/users/26cf87bd28c64f4c9310d680510a80d4/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:40.011502 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011abf33-a97c-488b-95ac-074c6b6ade3b None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:40.012076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011abf33-a97c-488b-95ac-074c6b6ade3b None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:40.012076 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011abf33-a97c-488b-95ac-074c6b6ade3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:40.029406 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:40.070332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-011abf33-a97c-488b-95ac-074c6b6ade3b None None] MFA Rules not processed for user `26cf87bd28c64f4c9310d680510a80d4`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:40.071998 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-011abf33-a97c-488b-95ac-074c6b6ade3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:40.172209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1439/2875] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:40 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:40.190891 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:40.192979 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:40.287529 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0d47ff-340d-4233-90c7-eac719fe5a6c admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:40.293265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:40.293889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:40.294196 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:40.294461 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:40.358632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:40.360431 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:40.370629 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1437/2876] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:40 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 180 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:41.385407 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2d8aac-ab4e-422d-a821-58830f80a22c None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:41.385976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2d8aac-ab4e-422d-a821-58830f80a22c None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:41.385976 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e2d8aac-ab4e-422d-a821-58830f80a22c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:41.406401 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8e2d8aac-ab4e-422d-a821-58830f80a22c None None] MFA Rules not processed for user `6ce84f7e891242c393d0ac5e97c8a9e2`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:41.407738 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8e2d8aac-ab4e-422d-a821-58830f80a22c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:41.415702 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1440/2877] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:41 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:50.366548 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=58 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:51.400554 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=34 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:58.808372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:58.878327 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56486179-3dbb-4aed-ba3a-1a19a5a58958 service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:58.882985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:58.883604 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:58.883871 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:58.884130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] PATH_INFO: `/v3/users/8b1e13586ae54909adce04ca214f1365` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:58.885242 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b1e13586ae54909adce04ca214f1365)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:58.887195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:58.938715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b1e13586ae54909adce04ca214f1365'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:58.945767 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b1e13586ae54909adce04ca214f1365'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:58.946442 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b1e13586ae54909adce04ca214f1365'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:58.947089 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b1e13586ae54909adce04ca214f1365'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:58.956581 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b1e13586ae54909adce04ca214f1365'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:58.976858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1438/2878] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:25:58 2022] DELETE /identity/v3/users/8b1e13586ae54909adce04ca214f1365 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:58.981582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:58.982121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:58.982121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.002671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:59.003872 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:59.078728 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1441/2879] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:59.089165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.090786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:59.151337 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a37e52-12ca-4644-85b1-fcc916071ff8 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.155706 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4852d4c9-e571-4270-b87e-313bae133040 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.156311 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4852d4c9-e571-4270-b87e-313bae133040 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.156584 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4852d4c9-e571-4270-b87e-313bae133040 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.156852 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4852d4c9-e571-4270-b87e-313bae133040 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.167405 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4852d4c9-e571-4270-b87e-313bae133040 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.168858 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4852d4c9-e571-4270-b87e-313bae133040 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:59.178646 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1439/2880] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:25:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:25:59.449583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.451014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:59.465599 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-779c4d02-e17d-4d4d-9bfe-8971677c697e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.471925 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.472684 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.473011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.473189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] PATH_INFO: `/v3/projects/c25e1b1ada2349a896c46d992833023e` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.474583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c25e1b1ada2349a896c46d992833023e)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.476643 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:59.526480 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c25e1b1ada2349a896c46d992833023e', 'request_id': 'req-ddac19ff-9beb-4860-85d6-62c0250cf211'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:59.535427 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c25e1b1ada2349a896c46d992833023e', 'request_id': 'req-ddac19ff-9beb-4860-85d6-62c0250cf211'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:59.542454 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c25e1b1ada2349a896c46d992833023e', 'request_id': 'req-ddac19ff-9beb-4860-85d6-62c0250cf211'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:59.553525 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] The token cache is being invalidate because project c25e1b1ada2349a896c46d992833023e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:25:59.553812 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:59.554583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ddac19ff-9beb-4860-85d6-62c0250cf211 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:25:59.556152 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1442/2881] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:25:59 2022] DELETE /identity/v3/projects/c25e1b1ada2349a896c46d992833023e => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:25:59.561420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64939215-9bc8-4277-a684-10cd6bef68dd None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.561682 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64939215-9bc8-4277-a684-10cd6bef68dd None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.561976 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64939215-9bc8-4277-a684-10cd6bef68dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.582087 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-64939215-9bc8-4277-a684-10cd6bef68dd None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:59.583198 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-64939215-9bc8-4277-a684-10cd6bef68dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:59.600463 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:25:59.645934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1440/2882] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:25:59.655688 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4956cd8-3f07-4a64-b7f8-dfa6c775072e None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.655977 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4956cd8-3f07-4a64-b7f8-dfa6c775072e None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.656254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4956cd8-3f07-4a64-b7f8-dfa6c775072e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.676064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-d4956cd8-3f07-4a64-b7f8-dfa6c775072e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:25:59.677284 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d4956cd8-3f07-4a64-b7f8-dfa6c775072e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:25:59.685484 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1443/2883] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:25:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:25:59.689964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64939215-9bc8-4277-a684-10cd6bef68dd None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.699863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64939215-9bc8-4277-a684-10cd6bef68dd None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.705326 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.705916 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.706228 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.706536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] PATH_INFO: `/v3/domains` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.707154 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81683) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:25:59.707454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.709058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:59.721003 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1441/2884] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:25:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:25:59.725580 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4956cd8-3f07-4a64-b7f8-dfa6c775072e None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.735569 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4956cd8-3f07-4a64-b7f8-dfa6c775072e None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.741608 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.742242 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.742514 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.742775 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.743415 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.745936 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:59.778606 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1444/2885] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:25:59 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:59.782761 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.793534 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6ec3f2-45f1-4344-ae48-6a6d1b759f3d admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.854829 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.856181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.856455 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.857062 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] PATH_INFO: `/v3/users` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.857424 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.859368 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:59.883255 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1442/2886] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:25:59 2022] POST /identity/v3/users => generated 288 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:25:59.887801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.898855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e6d993c-5c83-4503-99ce-38cd7f533ee2 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.904692 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.905330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.905607 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.905875 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.906615 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.908326 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:59.919623 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1445/2887] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:25:59.924164 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.934673 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860054bf-da1a-4474-814b-827d31d162c9 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.939367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4d8068-7267-4691-bddd-e1530f3e1a04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.940009 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d8068-7267-4691-bddd-e1530f3e1a04 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.940384 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d8068-7267-4691-bddd-e1530f3e1a04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.940598 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4d8068-7267-4691-bddd-e1530f3e1a04 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.941179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d8068-7267-4691-bddd-e1530f3e1a04 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.942865 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4d8068-7267-4691-bddd-e1530f3e1a04 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:25:59.954057 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1443/2888] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:25:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 29 02:25:59.958231 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:25:59.968269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-addf5b39-c8a9-4b5b-99e6-10aa029bae06 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:25:59.973048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:25:59.973682 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:25:59.973953 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:25:59.974255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] PATH_INFO: `/v3/projects/dc905e15987d46eea284f353199a068a/users/2878911f759e4395a0381a6eb28ad8f5/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:25:59.989358 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] RBAC: Authorizing `identity:create_grant(project_id=dc905e15987d46eea284f353199a068a, user_id=2878911f759e4395a0381a6eb28ad8f5, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:25:59.992425 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:00.007802 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1446/2889] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:25:59 2022] PUT /identity/v3/projects/dc905e15987d46eea284f353199a068a/users/2878911f759e4395a0381a6eb28ad8f5/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:26:00.018292 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de3492-4376-41ed-99aa-f30dac13ca04 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:00.018575 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de3492-4376-41ed-99aa-f30dac13ca04 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:00.018848 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0de3492-4376-41ed-99aa-f30dac13ca04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:00.050588 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a0de3492-4376-41ed-99aa-f30dac13ca04 None None] MFA Rules not processed for user `2878911f759e4395a0381a6eb28ad8f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:00.051806 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a0de3492-4376-41ed-99aa-f30dac13ca04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:00.121837 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1444/2890] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:00 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:00.132742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:00.134795 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:00.157971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:00.195207 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b24b4a23-b985-43fd-9e2b-35b06793bf2a admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:00.199133 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2457d879-355c-48f7-b18e-4b3bfa920ec2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:00.199732 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457d879-355c-48f7-b18e-4b3bfa920ec2 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:00.200004 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457d879-355c-48f7-b18e-4b3bfa920ec2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:00.200266 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2457d879-355c-48f7-b18e-4b3bfa920ec2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:00.208672 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2457d879-355c-48f7-b18e-4b3bfa920ec2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:00.210219 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2457d879-355c-48f7-b18e-4b3bfa920ec2 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:00.219408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1447/2891] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:26:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:26:02.671045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:02.671619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:02.671619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:02.692374 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] MFA Rules not processed for user `2878911f759e4395a0381a6eb28ad8f5`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:02.693576 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:02.701543 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1445/2892] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:26:10.215056 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=17 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:12.687188 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:INSERT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:18.171398 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa30213-e375-432d-99a8-35275ce3b6cd None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:18.171893 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa30213-e375-432d-99a8-35275ce3b6cd None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:18.172006 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa30213-e375-432d-99a8-35275ce3b6cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:18.192032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-8fa30213-e375-432d-99a8-35275ce3b6cd None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:18.193218 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-8fa30213-e375-432d-99a8-35275ce3b6cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:18.253762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1448/2893] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:26:18 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:18.276664 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:18.278135 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:18.346854 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34dd7740-2dbe-48eb-ac8a-5ab652613f58 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:18.351932 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:18.352537 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:18.352889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:18.353155 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:18.361267 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:18.362894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:18.371377 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1446/2894] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:26:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:26:18.930997 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cde4d63-ff9a-4b51-9f55-3f1f29e10ac8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:18.931281 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cde4d63-ff9a-4b51-9f55-3f1f29e10ac8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:18.931555 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cde4d63-ff9a-4b51-9f55-3f1f29e10ac8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:18.951121 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-4cde4d63-ff9a-4b51-9f55-3f1f29e10ac8 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:18.952478 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-4cde4d63-ff9a-4b51-9f55-3f1f29e10ac8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:18.960881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1449/2895] 10.222.0.61 () {58 vars in 967 bytes} [Thu Sep 29 02:26:18 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 29 02:26:18.977352 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:18.994845 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9474080f-bb92-4ff7-aea9-db86525d37fa service swift] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:19.003758 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:19.004619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:19.005020 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:19.005315 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:19.015133 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:19.016650 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:19.024850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1447/2896] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:26:18 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:26:19.279261 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40275d14-e1ee-47e8-bfb7-931fab1b8137 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:19.279752 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40275d14-e1ee-47e8-bfb7-931fab1b8137 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:19.279752 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40275d14-e1ee-47e8-bfb7-931fab1b8137 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:19.300889 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-40275d14-e1ee-47e8-bfb7-931fab1b8137 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:19.302102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40275d14-e1ee-47e8-bfb7-931fab1b8137 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:19.310919 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1450/2897] 10.222.0.61 () {58 vars in 1009 bytes} [Thu Sep 29 02:26:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:19.323585 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:19.338653 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c67098-ca7f-467b-a924-2745b7833932 service swift] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:19.345021 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:19.345626 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:19.345912 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:19.346222 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:19.355130 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:19.356675 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:19.365454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1448/2898] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:26:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:26:19.718779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0595354e-cad7-48a5-95d9-85386addaf51 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:19.719061 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0595354e-cad7-48a5-95d9-85386addaf51 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:19.719363 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0595354e-cad7-48a5-95d9-85386addaf51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:19.740209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-0595354e-cad7-48a5-95d9-85386addaf51 None None] MFA Rules not processed for user `d505d9008c9540659a3de626874c2457`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:19.741890 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-0595354e-cad7-48a5-95d9-85386addaf51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:19.750087 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1451/2899] 10.222.0.61 () {58 vars in 967 bytes} [Thu Sep 29 02:26:19 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:19.765347 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:19.780619 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fafe79e-d6c6-4222-86d2-68060949f026 service swift] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:19.788102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eb1979e-b775-4d48-bfea-8ffa18619d34 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '34b928e9fe314d0fb61122b7a558c1b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:19.788654 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1979e-b775-4d48-bfea-8ffa18619d34 service swift] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:19.788934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1979e-b775-4d48-bfea-8ffa18619d34 service swift] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:19.789200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eb1979e-b775-4d48-bfea-8ffa18619d34 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:19.797831 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb1979e-b775-4d48-bfea-8ffa18619d34 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:19.799481 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2eb1979e-b775-4d48-bfea-8ffa18619d34 service swift] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:19.813757 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1449/2900] 10.222.0.61 () {60 vars in 1384 bytes} [Thu Sep 29 02:26:19 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 29 02:26:22.165755 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0595354e-cad7-48a5-95d9-85386addaf51 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:22.232291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0595354e-cad7-48a5-95d9-85386addaf51 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:22.237662 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:22.238654 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:22.238921 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:22.239189 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] PATH_INFO: `/v3/users/2878911f759e4395a0381a6eb28ad8f5` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:22.240403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2878911f759e4395a0381a6eb28ad8f5)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:22.242612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:22.295614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2878911f759e4395a0381a6eb28ad8f5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.302721 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2878911f759e4395a0381a6eb28ad8f5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.303381 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2878911f759e4395a0381a6eb28ad8f5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.304074 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2878911f759e4395a0381a6eb28ad8f5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.312762 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2878911f759e4395a0381a6eb28ad8f5'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.331503 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1452/2901] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:26:22 2022] DELETE /identity/v3/users/2878911f759e4395a0381a6eb28ad8f5 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:26:22.336232 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:22.336742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:22.336870 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:22.357052 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:22.358310 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:22.420006 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1450/2902] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:22.431176 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:22.432648 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:22.439483 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:22.495718 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f411555-708f-4894-9789-091bc1b79119 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:22.499704 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92336866-7953-4989-8129-652676d832f0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:22.500302 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92336866-7953-4989-8129-652676d832f0 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:22.500578 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92336866-7953-4989-8129-652676d832f0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:22.500843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92336866-7953-4989-8129-652676d832f0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:22.510915 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92336866-7953-4989-8129-652676d832f0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:22.512402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92336866-7953-4989-8129-652676d832f0 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:22.522385 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1453/2903] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:26:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:26:22.705472 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:22.706985 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:22.719715 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30974a8-77c2-44de-9ba5-2b40a2b5e5cf None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:22.724695 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:22.725334 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:22.725645 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:22.725929 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] PATH_INFO: `/v3/projects/dc905e15987d46eea284f353199a068a` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:22.727179 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] RBAC: Authorizing `identity:delete_project(project_id=dc905e15987d46eea284f353199a068a)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:22.729102 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:22.777322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:22.780423 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dc905e15987d46eea284f353199a068a', 'request_id': 'req-213b993c-643c-4e31-abd5-905a87c9487a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.789542 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dc905e15987d46eea284f353199a068a', 'request_id': 'req-213b993c-643c-4e31-abd5-905a87c9487a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.796078 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dc905e15987d46eea284f353199a068a', 'request_id': 'req-213b993c-643c-4e31-abd5-905a87c9487a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.806413 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] The token cache is being invalidate because project dc905e15987d46eea284f353199a068a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:26:22.806694 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.807322 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-213b993c-643c-4e31-abd5-905a87c9487a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:26:22.808793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1451/2904] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:26:22 2022] DELETE /identity/v3/projects/dc905e15987d46eea284f353199a068a => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:26:22.814164 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504dead-31d0-42f9-a9cf-f36c5ddd44c8 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:22.814442 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504dead-31d0-42f9-a9cf-f36c5ddd44c8 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:22.814712 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7504dead-31d0-42f9-a9cf-f36c5ddd44c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:22.836064 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-7504dead-31d0-42f9-a9cf-f36c5ddd44c8 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:22.837268 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7504dead-31d0-42f9-a9cf-f36c5ddd44c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:22.904377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1454/2905] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:22.912547 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a905c351-c821-49a6-88e4-5cdda5dfa55e None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:22.912850 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a905c351-c821-49a6-88e4-5cdda5dfa55e None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:22.913094 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a905c351-c821-49a6-88e4-5cdda5dfa55e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:22.933359 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-a905c351-c821-49a6-88e4-5cdda5dfa55e None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:22.934590 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-a905c351-c821-49a6-88e4-5cdda5dfa55e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:22.943316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1452/2906] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:22.947811 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504dead-31d0-42f9-a9cf-f36c5ddd44c8 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:22.958417 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7504dead-31d0-42f9-a9cf-f36c5ddd44c8 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:22.964476 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:22.965091 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:22.965362 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:22.965629 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] PATH_INFO: `/v3/domains` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:22.966291 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81684) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 29 02:26:22.966595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:22.968174 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:22.980187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1455/2907] 10.222.0.61 () {58 vars in 1153 bytes} [Thu Sep 29 02:26:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 29 02:26:22.984557 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a905c351-c821-49a6-88e4-5cdda5dfa55e None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:22.994998 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a905c351-c821-49a6-88e4-5cdda5dfa55e None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:23.000844 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:23.001454 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:23.001730 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:23.002018 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:23.002636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:23.004562 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:23.034790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1453/2908] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:26:22 2022] POST /identity/v3/projects => generated 379 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:26:23.039333 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:23.049927 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-707fc33e-a66f-49ba-88c4-2a317677db2e admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:23.120965 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:23.121582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:23.121855 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:23.122156 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:23.122803 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:23.124797 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:23.154582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1456/2909] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:26:23 2022] POST /identity/v3/users => generated 290 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:26:23.155797 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:23.167037 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8a103d2-0cc1-41f0-820b-2c3e3eed120e admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:23.172341 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:23.172944 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:23.173218 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:23.173488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:23.174169 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:23.175786 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:23.187331 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1454/2910] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:26:23.193468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:23.203734 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1da4db8f-d7f6-452e-a60f-2c5c06a34226 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:23.209350 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8352f8d-6092-483e-a0b7-e392f57638cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:23.209937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8352f8d-6092-483e-a0b7-e392f57638cb admin admin] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:23.210251 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8352f8d-6092-483e-a0b7-e392f57638cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:23.210621 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8352f8d-6092-483e-a0b7-e392f57638cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:23.211269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8352f8d-6092-483e-a0b7-e392f57638cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:23.212858 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8352f8d-6092-483e-a0b7-e392f57638cb admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:23.223364 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1457/2911] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:26:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:26:23.228063 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:23.238264 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc8db78-24b3-4c06-865d-a49445d662c6 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:23.244855 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:23.245499 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] REQUEST_METHOD: `PUT` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:23.245783 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:23.246044 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] PATH_INFO: `/v3/projects/e57890af7876437d9fddfea4a536a0d7/users/bbb9be467f4947f4a203a3de6c37fa9a/roles/9a454063cf7f41f1aeab078ffae345e0` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:23.262785 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e57890af7876437d9fddfea4a536a0d7, user_id=bbb9be467f4947f4a203a3de6c37fa9a, role_id=9a454063cf7f41f1aeab078ffae345e0)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:23.265452 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:23.292915 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1455/2912] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:26:23 2022] PUT /identity/v3/projects/e57890af7876437d9fddfea4a536a0d7/users/bbb9be467f4947f4a203a3de6c37fa9a/roles/9a454063cf7f41f1aeab078ffae345e0 => generated 0 bytes in 65 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:26:23.298199 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c9ed4-2a2b-4f84-a147-46e5570ac771 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:23.298535 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c9ed4-2a2b-4f84-a147-46e5570ac771 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:23.298930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b9c9ed4-2a2b-4f84-a147-46e5570ac771 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:23.321016 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-1b9c9ed4-2a2b-4f84-a147-46e5570ac771 None None] MFA Rules not processed for user `4aa345955ed042db9837b36c3d6532b0`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:23.322285 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-1b9c9ed4-2a2b-4f84-a147-46e5570ac771 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:23.331856 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:23.390789 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1458/2913] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:23.401484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:23.404433 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:23.473747 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d072e1-32b9-4b72-b6b9-184daf095998 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:23.478317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:23.478925 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:23.479195 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:23.479460 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:23.489064 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:23.490536 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:23.500001 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1456/2914] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:26:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:26:27.283636 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df2508-f9b1-4de8-a039-19d16edafc31 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:27.284125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df2508-f9b1-4de8-a039-19d16edafc31 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:27.284334 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27df2508-f9b1-4de8-a039-19d16edafc31 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:27.314876 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-27df2508-f9b1-4de8-a039-19d16edafc31 None None] MFA Rules not processed for user `bbb9be467f4947f4a203a3de6c37fa9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:27.316209 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-27df2508-f9b1-4de8-a039-19d16edafc31 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:27.389582 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1459/2915] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:27 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:27.394317 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:27.404934 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d4ad8dd-28e3-43dd-83a3-0595690fc7ff service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:27.409488 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:27.410247 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:27.410399 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:27.410666 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] PATH_INFO: `/v3/projects` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:27.411294 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:27.413265 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:27.445538 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1457/2916] 10.222.0.61 () {60 vars in 1153 bytes} [Thu Sep 29 02:26:27 2022] POST /identity/v3/projects => generated 379 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 29 02:26:27.446887 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df2508-f9b1-4de8-a039-19d16edafc31 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:27.460782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27df2508-f9b1-4de8-a039-19d16edafc31 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:27.537169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:27.538447 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:27.538668 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:27.538938 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] PATH_INFO: `/v3/users` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:27.539583 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:27.541534 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:27.571777 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1460/2917] 10.222.0.61 () {60 vars in 1144 bytes} [Thu Sep 29 02:26:27 2022] POST /identity/v3/users => generated 296 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 29 02:26:27.575800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:27.587641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2608065-156e-475f-97fd-9bcc1a472b8f admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:27.592526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:27.593256 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:27.593478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:27.593962 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:27.594639 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:27.596637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:27.608930 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1458/2918] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:26:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:26:27.613542 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:27.624640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-163ad72a-fe70-4b50-9ae8-77115a6e3c83 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:27.629716 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:27.630440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:27.630713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:27.631022 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] PATH_INFO: `/v3/projects/0ec6b7262ed54e9db3ef78fa18f2ef76/users/2930bb46c8e74a4e99a473930f520502/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:27.648200 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ec6b7262ed54e9db3ef78fa18f2ef76, user_id=2930bb46c8e74a4e99a473930f520502, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:27.651180 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:27.673465 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1461/2919] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:26:27 2022] PUT /identity/v3/projects/0ec6b7262ed54e9db3ef78fa18f2ef76/users/2930bb46c8e74a4e99a473930f520502/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:26:27.677623 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:27.690316 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d0a2b0d-9768-4816-b73a-bceb34d832cc admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:27.736923 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:27.759054 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d43ed8d0-cbb7-4c53-8395-aa02070ffd8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:27.759692 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43ed8d0-cbb7-4c53-8395-aa02070ffd8e admin admin] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:27.759959 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43ed8d0-cbb7-4c53-8395-aa02070ffd8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:27.760219 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d43ed8d0-cbb7-4c53-8395-aa02070ffd8e admin admin] PATH_INFO: `/v3/roles` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:27.760860 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43ed8d0-cbb7-4c53-8395-aa02070ffd8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:27.762484 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d43ed8d0-cbb7-4c53-8395-aa02070ffd8e admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:27.778779 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:27.782086 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1459/2920] 10.222.0.61 () {58 vars in 1122 bytes} [Thu Sep 29 02:26:27 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 29 02:26:27.790032 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8dc6b0b3-d200-40bc-aec2-cd86cac606c0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:27.795081 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:27.795766 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] REQUEST_METHOD: `PUT` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:27.796039 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:27.796306 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] PATH_INFO: `/v3/projects/0ec6b7262ed54e9db3ef78fa18f2ef76/users/2930bb46c8e74a4e99a473930f520502/roles/69aa90da9c204a0fbb53f7289b184185` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:27.798540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ec6b7262ed54e9db3ef78fa18f2ef76, user_id=2930bb46c8e74a4e99a473930f520502, role_id=69aa90da9c204a0fbb53f7289b184185)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:27.801115 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:27.807244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:27.821658 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1462/2921] 10.222.0.61 () {60 vars in 1483 bytes} [Thu Sep 29 02:26:27 2022] PUT /identity/v3/projects/0ec6b7262ed54e9db3ef78fa18f2ef76/users/2930bb46c8e74a4e99a473930f520502/roles/69aa90da9c204a0fbb53f7289b184185 => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:26:32.794349 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:32.794833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:32.795191 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:32.826900 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] MFA Rules not processed for user `2930bb46c8e74a4e99a473930f520502`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:32.829110 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:32.911782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1460/2922] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:32 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:26:32.929106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:32.931373 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:33.012388 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5ac4a43-3ffb-4665-a845-ca9039f5690b admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:33.017968 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7638d450-f65f-4279-8b21-e941206d9af4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:33.018619 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638d450-f65f-4279-8b21-e941206d9af4 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:33.018881 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638d450-f65f-4279-8b21-e941206d9af4 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:33.019157 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7638d450-f65f-4279-8b21-e941206d9af4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:33.073970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7638d450-f65f-4279-8b21-e941206d9af4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:33.075596 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7638d450-f65f-4279-8b21-e941206d9af4 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:33.085448 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1463/2923] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:26:32 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:26:42.907867 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=45 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:43.080990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=58 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:26:51.905205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:26:51.907015 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:51.978412 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f49c908f-eb4f-48b3-b24a-e15fcb77e627 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:26:51.984659 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:26:51.985263 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:51.985533 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:51.985796 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:51.987863 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:51.999738 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:26:52.001305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:26:52.011416 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1461/2924] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:26:51 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:26:54.822289 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] REQUEST_METHOD: `POST` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:26:54.823260 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:26:54.823513 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:26:54.851357 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.auth.core [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] MFA Rules not processed for user `bbb9be467f4947f4a203a3de6c37fa9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81684) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:26:54.853146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:26:54.861723 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1464/2925] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:26:54 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:27:02.006901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=32 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:27:04.841574 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=3 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:27:10.483022 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:27:10.495250 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d12b72-6251-410f-93d4-42d8628396b4 service placement] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:27:10.500549 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:27:10.501286 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:27:10.501569 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:27:10.501838 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:27:10.503632 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:10.518168 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:27:10.520060 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:27:10.540304 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1462/2926] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:27:10 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:27:13.687394 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:27:13.688807 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:13.755108 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f285b3a3-6a2d-4d0a-949b-7cdd4536e740 None None] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:27:13.759920 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:27:13.760560 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:27:13.760832 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:27:13.761104 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] PATH_INFO: `/v3/users/6ce84f7e891242c393d0ac5e97c8a9e2` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:27:13.762236 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=6ce84f7e891242c393d0ac5e97c8a9e2)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:27:13.764151 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:27:13.825290 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6ce84f7e891242c393d0ac5e97c8a9e2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:13.833051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6ce84f7e891242c393d0ac5e97c8a9e2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:13.834630 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6ce84f7e891242c393d0ac5e97c8a9e2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:13.835403 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6ce84f7e891242c393d0ac5e97c8a9e2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:13.845012 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6ce84f7e891242c393d0ac5e97c8a9e2'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:13.868468 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1465/2927] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:27:13 2022] DELETE /identity/v3/users/6ce84f7e891242c393d0ac5e97c8a9e2 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:27:14.044378 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:27:14.059984 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5085517-f69b-4565-9267-662c13e0db4b service placement] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:27:14.123975 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:27:14.124521 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:27:14.124793 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:27:14.125058 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] PATH_INFO: `/v3/projects/a2f2d778c35d4a90a28aeef64fefa8fb` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:27:14.126200 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a2f2d778c35d4a90a28aeef64fefa8fb)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:27:14.128123 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:27:14.176004 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a2f2d778c35d4a90a28aeef64fefa8fb', 'request_id': 'req-9b65d325-085b-4c94-acf0-25a59923495a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.183951 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a2f2d778c35d4a90a28aeef64fefa8fb', 'request_id': 'req-9b65d325-085b-4c94-acf0-25a59923495a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.190144 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a2f2d778c35d4a90a28aeef64fefa8fb', 'request_id': 'req-9b65d325-085b-4c94-acf0-25a59923495a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.200215 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] The token cache is being invalidate because project a2f2d778c35d4a90a28aeef64fefa8fb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:27:14.200492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.201109 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.202636 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1463/2928] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:27:14 2022] DELETE /identity/v3/projects/a2f2d778c35d4a90a28aeef64fefa8fb => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:27:14.207393 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:27:14.209980 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:14.228823 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd23507-8cdd-4b2e-aded-da0199f361ee admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:27:14.289395 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:27:14.290051 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:27:14.290342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:27:14.290614 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] PATH_INFO: `/v3/users/26cf87bd28c64f4c9310d680510a80d4` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:27:14.291764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] RBAC: Authorizing `identity:delete_user(user_id=26cf87bd28c64f4c9310d680510a80d4)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:27:14.293695 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:27:14.317930 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:27:14.352001 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '26cf87bd28c64f4c9310d680510a80d4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.363740 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '26cf87bd28c64f4c9310d680510a80d4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.364612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '26cf87bd28c64f4c9310d680510a80d4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.365386 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '26cf87bd28c64f4c9310d680510a80d4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.375898 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '26cf87bd28c64f4c9310d680510a80d4'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.398421 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1466/2929] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:27:14 2022] DELETE /identity/v3/users/26cf87bd28c64f4c9310d680510a80d4 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:27:14.586892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:27:14.588612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:14.602603 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b65d325-085b-4c94-acf0-25a59923495a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:27:14.660842 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:27:14.661504 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:27:14.661782 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:27:14.662084 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] PATH_INFO: `/v3/projects/6b66385ef54a4c58af4bb356e1550354` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:27:14.663205 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6b66385ef54a4c58af4bb356e1550354)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:27:14.665120 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:27:14.685964 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:27:14.714402 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6b66385ef54a4c58af4bb356e1550354', 'request_id': 'req-aca529bc-55bd-4343-9468-85acfe741969'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.723210 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6b66385ef54a4c58af4bb356e1550354', 'request_id': 'req-aca529bc-55bd-4343-9468-85acfe741969'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.730118 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6b66385ef54a4c58af4bb356e1550354', 'request_id': 'req-aca529bc-55bd-4343-9468-85acfe741969'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.740551 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] The token cache is being invalidate because project 6b66385ef54a4c58af4bb356e1550354 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81683) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:27:14.740772 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.741404 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:27:14.742924 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1464/2930] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:27:14 2022] DELETE /identity/v3/projects/6b66385ef54a4c58af4bb356e1550354 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:27:19.485440 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:27:19.487000 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:19.550620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed17743d-bb13-4ae4-9c6c-5a0016a4d772 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:27:19.602819 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:27:19.603434 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:27:19.603707 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:27:19.603975 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:27:19.605937 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:19.672687 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:27:19.674060 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:27:19.683869 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1467/2931] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:27:19 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:27:21.308667 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:27:21.310248 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:21.396889 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aca529bc-55bd-4343-9468-85acfe741969 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:27:21.402981 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:27:21.403593 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:27:21.403866 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:27:21.404132 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:27:21.406025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:27:21.419129 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:27:21.420726 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:27:21.431559 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1465/2932] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:27:21 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:27:29.679801 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:DELETE=7,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:27:31.427371 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=50,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:28:55.337639 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:28:55.339377 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:28:55.351102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa14e036-a14f-4579-91ba-d40c79640c48 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:28:55.355674 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:28:55.356269 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:28:55.356540 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:28:55.356804 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:28:55.358733 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:28:55.369759 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:28:55.371211 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:28:55.379383 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1468/2933] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:28:55 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:29:05.375926 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:29:35.941591 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:29:35.952028 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2884cfe-83cd-43bc-9fe3-0433c210c4ab service nova] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:29:35.956293 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4cfd815-faad-42cb-8f66-e23fc13e3cc4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:29:35.956905 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cfd815-faad-42cb-8f66-e23fc13e3cc4 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:29:35.957181 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cfd815-faad-42cb-8f66-e23fc13e3cc4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:29:35.957450 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4cfd815-faad-42cb-8f66-e23fc13e3cc4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:29:35.959437 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-b4cfd815-faad-42cb-8f66-e23fc13e3cc4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:29:35.973683 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cfd815-faad-42cb-8f66-e23fc13e3cc4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:29:35.975245 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4cfd815-faad-42cb-8f66-e23fc13e3cc4 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:29:35.986406 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1466/2934] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:29:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:29:38.946330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:29:38.964397 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a21d0f-d4fc-4118-9fb5-57697ad8be48 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:29:38.970213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:29:38.971460 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:29:38.971739 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:29:38.972014 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:29:38.974339 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:29:38.989197 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:29:38.990843 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:29:39.003924 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1469/2935] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:29:38 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 58 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:29:40.761612 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:29:40.762122 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:29:40.762236 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:29:40.781329 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] MFA Rules not processed for user `bbb9be467f4947f4a203a3de6c37fa9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:29:40.782546 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:29:40.837668 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1467/2936] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:29:40 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:29:41.169491 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:29:41.179914 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40a3bbc0-8d4e-43b2-be4d-d235330b8497 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:29:41.184671 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:29:41.185258 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:29:41.185528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:29:41.185790 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:29:41.188332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:29:41.205640 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:29:41.207105 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:29:41.216017 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1470/2937] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:29:41 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:29:41.696410 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:29:41.698367 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:29:41.774277 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fe548c-315d-43a3-ad64-3fb79d1452e7 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:29:41.779173 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:29:41.779774 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:29:41.780045 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:29:41.780309 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:29:41.782025 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:29:41.793773 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:29:41.795388 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:29:41.805512 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1468/2938] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:29:41 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:29:51.211729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=12 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:29:51.801478 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=63 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:29:57.204634 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:29:57.220713 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a33aa919-2806-44fb-a7d2-6ecf1566408d service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:29:57.227134 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:29:57.227747 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:29:57.228023 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:29:57.228330 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:29:57.230342 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:29:57.243254 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:29:57.244935 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:29:57.256528 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1471/2939] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:29:57 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 53 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:30:07.251546 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=6 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:31:56.853271 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:31:56.919790 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4434e9b8-0e53-400d-a560-f26b90e05941 service placement] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:31:56.929440 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c3dc151-5b11-467e-abd4-54fc47b8001f service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:31:56.930320 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3dc151-5b11-467e-abd4-54fc47b8001f service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:31:56.930680 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3dc151-5b11-467e-abd4-54fc47b8001f service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:31:56.930901 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c3dc151-5b11-467e-abd4-54fc47b8001f service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:31:56.933734 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-5c3dc151-5b11-467e-abd4-54fc47b8001f service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:31:56.972565 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3dc151-5b11-467e-abd4-54fc47b8001f service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:31:56.974193 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c3dc151-5b11-467e-abd4-54fc47b8001f service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:31:56.985167 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1469/2940] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:31:56 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:32:04.201971 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:32:04.204612 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:32:04.337946 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37b1f78-06f7-4540-bc18-ba09f571f53a service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:32:04.343125 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '36c4d127aac743baaeb86157dbbeb689', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:32:04.343729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:32:04.344011 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:32:04.344257 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:32:04.346445 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:32:04.358408 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:32:04.359828 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:32:04.368699 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1472/2941] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:32:04 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 168 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:32:06.980104 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=36 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:32:14.364934 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=48 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:32:15.593945 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2490097b-778c-4a1f-a3b3-10ef942516e9 None None] REQUEST_METHOD: `POST` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:32:15.594496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2490097b-778c-4a1f-a3b3-10ef942516e9 None None] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:32:15.594496 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2490097b-778c-4a1f-a3b3-10ef942516e9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:32:15.613678 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.auth.core [None req-2490097b-778c-4a1f-a3b3-10ef942516e9 None None] MFA Rules not processed for user `bbb9be467f4947f4a203a3de6c37fa9a`. Rule list: `[]` (Enabled: `True`). {{(pid=81683) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 29 02:32:15.614892 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-2490097b-778c-4a1f-a3b3-10ef942516e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:32:15.625637 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1470/2942] 10.222.0.61 () {58 vars in 958 bytes} [Thu Sep 29 02:32:15 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 29 02:32:25.608486 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=3 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:32:39.140772 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:32:39.170347 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34a85bb-e6d4-4a42-a966-c888d4484235 service cinder] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:32:39.175950 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:32:39.176597 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:32:39.176851 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:32:39.177150 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:32:39.187750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:32:39.189194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:32:39.198879 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1473/2943] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:32:39 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:32:49.194764 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=10 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:33:57.560957 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2490097b-778c-4a1f-a3b3-10ef942516e9 None None] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:33:57.572314 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2490097b-778c-4a1f-a3b3-10ef942516e9 None None] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:33:57.576927 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:33:57.577526 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:33:57.577800 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:33:57.578093 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:33:57.586241 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:33:57.587681 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:33:57.596142 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1471/2944] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:33:57 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:34:07.592449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:34:44.458367 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:34:44.490402 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-424c7123-50e6-4ab4-a7d6-ce0db3872108 service neutron] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:34:44.497116 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:34:44.497691 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:34:44.497961 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:34:44.498262 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:34:44.508248 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:34:44.509839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:34:44.520885 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1474/2945] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:34:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 64 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:34:44.607105 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:34:44.616372 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535df07b-eb8a-4af6-923e-f0dae9b0043a service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:34:44.620833 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:34:44.621597 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:34:44.621714 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:34:44.621978 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:34:44.632956 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:34:44.634420 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:34:44.645570 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1472/2946] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:34:44 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:34:46.965146 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:34:46.978498 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ad39cd-815d-48a2-b8dc-a69e1564a579 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:34:46.984194 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'bc807d5f532b4e4c9953ba5e7eb878bb', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member', 'ResellerAdmin', 'service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:34:46.984782 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:34:46.985057 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:34:46.985327 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:34:46.995839 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:34:47.000119 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:34:47.011954 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1475/2947] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:34:46 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:34:54.641183 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=6 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:34:56.946936 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:34:56.956361 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3ce5599-da39-43fc-accf-631383aad973 service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:34:56.960297 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'cc2f041fe7194a3da199517b4cdbd265', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:34:56.960886 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:34:56.961160 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:34:56.961432 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:34:56.963305 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:34:57.006316 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=19 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:34:57.049742 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:34:57.051567 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:34:57.073418 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1473/2948] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:34:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:35:00.243579 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:35:00.245233 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:35:00.260319 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e174f96-74cb-4b18-a048-4d4b208aafd7 service nova] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:35:00.265595 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:35:00.266335 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] REQUEST_METHOD: `GET` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:35:00.266784 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:35:00.267244 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:35:00.269195 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:35:00.282106 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:35:00.283903 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:35:00.297187 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1476/2949] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:35:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:35:03.489269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:35:03.490720 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:35:03.505500 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b675285-95ad-4def-ba87-2653d85f0ebf service neutron] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:35:03.512453 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:35:03.513449 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:35:03.513729 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:35:03.514019 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] PATH_INFO: `/v3/users/bbb9be467f4947f4a203a3de6c37fa9a` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:35:03.515165 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] RBAC: Authorizing `identity:delete_user(user_id=bbb9be467f4947f4a203a3de6c37fa9a)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:35:03.517097 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:35:03.571737 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'bbb9be467f4947f4a203a3de6c37fa9a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.578974 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'bbb9be467f4947f4a203a3de6c37fa9a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.579620 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'bbb9be467f4947f4a203a3de6c37fa9a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.580207 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'bbb9be467f4947f4a203a3de6c37fa9a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.588482 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'bbb9be467f4947f4a203a3de6c37fa9a'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.606824 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1474/2950] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:35:03 2022] DELETE /identity/v3/users/bbb9be467f4947f4a203a3de6c37fa9a => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:35:03.785088 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:35:03.805048 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d070bff1-4e30-4339-bd45-434505844206 service placement] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:35:03.863577 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:35:03.864213 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:35:03.864488 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:35:03.864757 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] PATH_INFO: `/v3/projects/e57890af7876437d9fddfea4a536a0d7` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:35:03.865896 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e57890af7876437d9fddfea4a536a0d7)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:35:03.867874 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:35:03.918404 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e57890af7876437d9fddfea4a536a0d7', 'request_id': 'req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.926679 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e57890af7876437d9fddfea4a536a0d7', 'request_id': 'req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.933259 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e57890af7876437d9fddfea4a536a0d7', 'request_id': 'req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.943706 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] The token cache is being invalidate because project e57890af7876437d9fddfea4a536a0d7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:35:03.943990 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.944620 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:03.946160 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1477/2951] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:35:03 2022] DELETE /identity/v3/projects/e57890af7876437d9fddfea4a536a0d7 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:35:03.950358 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:35:03.951815 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:35:03.964762 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef0b62b-abf1-4cf7-8a91-2d7e83a84611 admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:35:03.997354 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:35:04.022641 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:35:04.023269 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:35:04.023540 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:35:04.023802 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] PATH_INFO: `/v3/users/2930bb46c8e74a4e99a473930f520502` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:35:04.024946 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=2930bb46c8e74a4e99a473930f520502)` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:35:04.026903 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:35:04.077894 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2930bb46c8e74a4e99a473930f520502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.085385 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2930bb46c8e74a4e99a473930f520502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.086136 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2930bb46c8e74a4e99a473930f520502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.086799 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2930bb46c8e74a4e99a473930f520502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.096811 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.notifications [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2930bb46c8e74a4e99a473930f520502'} {{(pid=81683) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.122674 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1475/2952] 10.222.0.61 () {58 vars in 1224 bytes} [Thu Sep 29 02:35:03 2022] DELETE /identity/v3/users/2930bb46c8e74a4e99a473930f520502 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:35:04.307970 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Authenticating user token {{(pid=81684) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:35:04.309765 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.fernet_utils [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81684) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:35:04.324255 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21131cc1-9d73-4dae-b4ac-6b4770f0d7f0 admin admin] Validating token access rules against request {{(pid=81684) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:35:04.383477 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4aa345955ed042db9837b36c3d6532b0', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'c6a6000fb0914b0390fa25e08205ee14', 'project_domain_id': 'default', 'roles': ['member', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81684) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:35:04.384102 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:35:04.384370 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] SCRIPT_NAME: `/identity` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:35:04.384638 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] PATH_INFO: `/v3/projects/0ec6b7262ed54e9db3ef78fa18f2ef76` {{(pid=81684) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:35:04.385750 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ec6b7262ed54e9db3ef78fa18f2ef76)` {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:35:04.387791 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] RBAC: Authorization granted {{(pid=81684) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:35:04.410742 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:35:04.438729 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ec6b7262ed54e9db3ef78fa18f2ef76', 'request_id': 'req-68fbe8d2-ef4d-41e5-99f7-9cada7268669'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.447169 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ec6b7262ed54e9db3ef78fa18f2ef76', 'request_id': 'req-68fbe8d2-ef4d-41e5-99f7-9cada7268669'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.454846 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ec6b7262ed54e9db3ef78fa18f2ef76', 'request_id': 'req-68fbe8d2-ef4d-41e5-99f7-9cada7268669'} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.467303 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] The token cache is being invalidate because project 0ec6b7262ed54e9db3ef78fa18f2ef76 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81684) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 29 02:35:04.467567 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.468332 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG keystone.notifications [None req-68fbe8d2-ef4d-41e5-99f7-9cada7268669 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81684) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 29 02:35:04.469891 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: [pid: 81684|app: 0|req: 1478/2953] 10.222.0.61 () {58 vars in 1233 bytes} [Thu Sep 29 02:35:04 2022] DELETE /identity/v3/projects/0ec6b7262ed54e9db3ef78fa18f2ef76 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 29 02:35:14.116763 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=36,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:35:14.463592 dv-cinder-733062-36-iscsi devstack@keystone.service[81684]: DEBUG dbcounter [-] [81684] Writing DB stats keystone:SELECT=16,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81684) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 29 02:37:02.828492 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Authenticating user token {{(pid=81683) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 29 02:37:02.830182 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:37:02.909586 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5a51e9-aef0-4210-9a00-88ba563c9d1a admin admin] Validating token access rules against request {{(pid=81683) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 29 02:37:02.976151 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27671c29-6fd8-46ec-8f3b-0137c5390a81 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '8083af87f4f742be8b769d8b0bb7ca6f', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae7c64039839431980d6f61231270abc', 'project_domain_id': 'default', 'roles': ['member', 'service', 'reader', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81683) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 29 02:37:02.976759 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27671c29-6fd8-46ec-8f3b-0137c5390a81 service placement] REQUEST_METHOD: `GET` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 29 02:37:02.977034 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27671c29-6fd8-46ec-8f3b-0137c5390a81 service placement] SCRIPT_NAME: `/identity` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 29 02:37:02.977288 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27671c29-6fd8-46ec-8f3b-0137c5390a81 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81683) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 29 02:37:02.979253 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.fernet_utils [None req-27671c29-6fd8-46ec-8f3b-0137c5390a81 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81683) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 29 02:37:02.991535 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27671c29-6fd8-46ec-8f3b-0137c5390a81 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 29 02:37:02.993126 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27671c29-6fd8-46ec-8f3b-0137c5390a81 service placement] RBAC: Authorization granted {{(pid=81683) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 29 02:37:03.003214 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: [pid: 81683|app: 0|req: 1476/2954] 10.222.0.61 () {60 vars in 1365 bytes} [Thu Sep 29 02:37:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 29 02:37:12.999075 dv-cinder-733062-36-iscsi devstack@keystone.service[81683]: DEBUG dbcounter [-] [81683] Writing DB stats keystone:SELECT=58 {{(pid=81683) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}